I had this problem with all my sites Thursday last week. Robber had a site dropped Thursday this week as well.
See this thread
Just curious. If you had hundreds of pages already in the index for years, why did you decide to go with PFI?
I've read a number of posts here from people who have had bad experiences with Ink's PFI. Dropping all your pages is terrible. I'm wondering if those who stick strickly with the free indexing are experiencing any similar problems as you all.
BlueSky - I've HAD hundreds of pages indexed for free but also some PFI pages. Some sites have a mix of both free and PFI, some sites completely indexed with no PFI. It doesn't seem to make any difference if a site has PFI pages or not when it comes to the rest of the site being indexed for free, I haven't noticed any difference betweent hose sites with PFI and those that don't.
But now all my sites were dropped, those with PFI and those with no PFI at all. PFI ones are getting back in when they would have been refreshed anyway.
When I put a new site up I often PFI a few pages that I want in the index quickly. Then those I don't mind waiting a month or more to be indexed I leave to find their way in for free. A page can be up in a couple of days with PFI, but take over a month (maybe much more), for free, if in that time the page makes more than the cost of the PFI, then it's a good ROI.
I've had sites make several thousand dollars from just a few PFI pages before the rest of the site was indexed for free, this makes the PFI seem pretty cheap. I won't PFI any pages where the L$ listings dominate though.
I also have pages indexed as PFI if I want to make changes to those pages and get them re-indexed frequently. Special offer pages or pages where when a change is made you want that page up quickly.
Thanks or the information, and for pointing me to the other thread.
The fact that others are having the same problems with whole sites dropping out suggests to me unfortunately that Ink is indeed changing its algorithm, perhaps following from its acquisition by Yahoo. Could this be under way incrementally, so their index is changing from week to week? Could there be a purge of commerical/affiliate sites going on, unless you pay up?
I agree with what was said above, I have never before had any trouble with a combination of free and PFI pages.
The reason I use PFI is that it allows me to 1. test pages and 2. get pages on 'hot' topics up quickly. When fads like Segway, Italian charm bracelets or Jasket Power hit the news there is often little or no competition, so getting a page indexed and up within a few days almost guarantees big traffic.
However, for the site in question I relied mainly on the free traffic from MSN, and having now lost that traffic the site (which I have worked on daily for the last two years) is effectively dead in the water.
My PFI pages are being added back in, but no crawl for the ones that were in for free.
Trouble is, the PFI pages aren't getting the positions they did have for my keywords, they are way way down in the SERPS.
Callanish, be interesting to hear if your PFI pages get back in, and what positions they get compared to what they did have for your keywords.
I also did a little experiment, for a very popular and competative keyword I checked how many pages the top 10 in google had on Inktomi. You need hundreds of backlinks to appear that high. The most any of them had in Inktomi was 4 pages, most only had 2 pages listed in Inktomi and a couple had none. These are old established sites with hundreds of backlinks, I don't know how many pages they had in Inktomi in the past, but you would expect them to do pretty well given the number of backlinks and the age of the sites.
All very strange.
Now found some of my sites, PFI pages crawled 14th (and they have been on PFI for a long time), that are not listed on inktomi.
I've sent a support request to Webwurld asking if they can shed any light on the problem.
I'll report back the response I get.
Thanks for the info. I was considering PFI, but I'll wait on the sidelines for now and see how your stories turn out. I don't think it's right for SE's to completely drop sites like Ink's doing without saying a peep why.
Looking through the logs of the site that dropped Slurp has been looking around - yesterday and today, but not back in the index yet.
Slurp hasn't been around my pages yet apart from a few PFI ones.
Initial reply from Webwurld included the phrase
"its almost impossible to find out whether a page has been banned. Inktomi started banning pages in 6 month cycles about a year ago"
I've sent a reply back trying to find a specific reason why all my sites suddenly disappeared and spidering suddenly stopped, will post any replies and anything of interest like the above that comes out of it.
"Callanish, be interesting to hear if your PFI pages get back in, and what positions they get compared to what they did have for your keywords."
My PFI pages have never been out of Ink, and their traffic from MSN has stayed very similar. I'm sure if I paid 500 x $25 I'd get all the traffic back. But since no-one seems to know what Yahoo is going to do with Inktomi in the next few months that is quite a large risk.
I have found out that the Ink update cycle for free pages is about a month, so I can still hope my pages will get back in.
SeventiesMartin, I hope you can keep us posted with developments with your site.
All PFI pages have now been spidered, i just hope they get their positions back when they appear back in MSN, assuming they do get back into MSN.
Not heard any more from Webwurld. They did say they could contact Inktomi about any problems, buy couldn't talk about specific domains. I gave then my the IP address of my server, as it is all my domains on that server and only that server that were dropped.
Hopefully the delay is waiting for a reply from Inktomi. They were very quick with the first reply which was impressive, so I have faith in the support dept.
Will post when I know more.
Webwurld confirmed they have sent my questions to Inktomi and are waiting for a response.
Well, having had my free pages dropped from Ink this day last week, now my PFI pages have fallen out too.
Strangely, I can still find the pages in MSN by searching on the domain name, but for the keywords they pages must be buried so deep they may as well not exist.
This looks like an sudden editorial penalty, since the pages have not changed recently.
I guess I have joined the 'gone from Ink' club. I wish I had a clue why this is happening.
I think INK is the worst engine out there. The results are just not good. I can't find anything I'm looking for. All The Web is so much better and stable. Less spam, more results. Why in the world would Yahoo use INK over ATW?... Paid inclusion $. That's the only reason they would ever chose INK IMHO.
|In the absence of any comment about recent changes to the Ink algorithm, I assume this has been caused by my site being unavailable when the spider came round before the last index build. The site has been down twice for periods of up to eight hours in the last month. |
I don't think that could be the problem. I have a site thats been out in the wild since 3 years and it had got several thousand pages listed in ink, but the past fortnight, most of them have dropped leaving only a meagre hundreds to rot with no exposure.
Same thing has happened to a couple of my sites. I am sure this must have happened to many many sites, but since its not google, there isn't much discussion.
I just read that Yahoo is planning to ditch Google for Inktomi soon. If it uses this inktomi database, then I for one will never use it and so will many others. If my 3 year old site which has a good placement and all it's pages crawled in google suddenly lose out in inktomi, I have every reason to believe that the problem lies not with my site, but with inktomi ;) Similarly, tons of other sites have missing pages as I checked.
OK, there seems to be an Ink purge going on - day by day and week by week, free pages are dropping out.
(I agree, if this were Google the whole webmaster community would be jumping up and down.)
But why would they purge or degrade inclusion of their PFI pages? This makes no sense to me. Unless PFI in Ink has no future anyway - perhaps Yahoo isn't interested in carrying PFI pages, and it's now calling the shots for the Ink index, or requiring it to have less affiliate/commercial listings.
No response from Inktomi yet.
I've noticed that my Inktomi PFI pages are appearing again on pure Inktomi search, but only a maximum of 2 on MSN, hotbot etc.
I checked on sites in the top ten on google for a very popular search phrase, to get these positions these sites have a lot of backinks and are well established. Again, pure search shows a fair number of pages, MSN only a few, usually 2.
It's almost as if the number of pages being picked up from Inktomi is being limited.
So, I pay for 8 pages of a site to be indexed, and only 2 appear on MSN etc.
One of my pages that is indexed until a few weeks ago was in the the top positions for the product it promotes. I went all the way through the index, nearly 900 results, and it's now in the bottom 10. The site immediatly above is a Russian p**n site with not a mention of the products name. Actually, a lot of the results above don't mention the product name but still appear in the results above my page.
I also noticed several other sites that were doing well for the product that are at the bottom of the SERPS as well.
Maybe they intended to change the Algo so PFI pages appear first to promote PFI, but got it the wrong way round and now they appear last, if only.
If PFI pages are not going to be included or are penalised, then I think they should not accept the order. To take your money knowing that you aren't going to be able to compete on even terms with other pages seems downright immoral to me.
I have 2 pages of my site with lyco's Paid inclusion (Fast and Inktomi).
Site has been running since April. Google has 154 search results for my domain name. (140 of my pages and a dozen or so incoming links)
Fast and Inktomi have the 2 PFI only
The only conclusion I can draw is that if you're using PFI, they cramp your free listings to force you to pay for more pages.
I feel penalized for being their customer.
|OK, there seems to be an Ink purge going on - day by day and week by week, free pages are dropping out |
Yep, back in June 71% of one of my site's traffic was from MSN, now it's down to 17% with about a 50% drop in real numbers from MSN. Real glad that the site got back in G's good graces at that time but that is now a very scary 50+%.
If webmasters are having a common experience of losing free pages and getting badly ranked PFI pages in Inktomi, we should, as a group, be able to work out why it is happening even if Inktomi won't tell us.
I assume this is happening because of a flag to human reviewer, who imposes a manual penalty or demotion on viewing the site. So if we brainstorm the possible reasons perhaps we could come to a likely set of reasons for a penalty.
I will suggest as wild or not so wild possiblities for flagging a site/demotion or removal of a site:
More than 200 characters in the meta keywords
Keywords which don't appear in the page body
More than 250 characters in the meta description
More than 120 characters in the title
Use of DMOZ (Open Directory) content or feed
More than 500 pages in the site
Multiple affilate links on a page
Use of white text, even on a colored background
'Adult' themed pages even if legitimate, eg adult personals
'Gay' themed pages even if legitimate, eg gay personals
Pages over 100k in size
Any more ideas?
I would hope they aren't looking at things this way but here goes:
Hyphens in domain name
Running PPC search results
Lack of a dmoz entry
I think they look for keyword spamming among industries that are most often spammed. I don't think they look specifically for affiliate sites, it's just that affiliate sites often push the envelope and wind up stepping over their secret line. After all, Inktomi gives webmasters the PFI 48-hour update tool to help them rank better. Once the webmaster learns how to rank well, they may well have crossed the line into keyword spamming and get sacked.
It's kind of like giving a wino a bottle of booze, then coming back and arresting him for public drunkeness.
My logs and experience indicate that they sack your whole site, free or PFI or both, based on a visit to a single page from a human editor. It also apprears the human editor visit is preceeded within hours or a few days by an algorithmic scan and scoring of a single page.
Shortly following the human editor visit, the free pages are totally removed from the index. A few days later the PFI pages drop drastically in ranking but remain in the index and continue to be spidered until the one-year payment period expires.
I did see once where the free pages disappeared but the PFI pages remained at good ranking for another six months.
Check your raw logs for a visit from idev2.inktomi.com/reltest/ followed later by a visit from peyote.inktomi.com/
It looks like to do well with Ink you might have to take up cloaking. Lets hear some pointers from the cloaking folks.
Looks like Mayor has hit the nail on the head - just checked the logs and we had:
The last part of the URL looks like it could have the reviewers name in it.
So, a human review has led to a ban by the looks of it. But how to go about getting the ban lifted since Inktomi seem pretty hard to contact.
Since it doesn't seem to be an algorithmic penalty I guess theres not much hope for getting indexed again.
Has anyone had their pages re-included and has anyone ever managed to speak to someone that could shed some light?
It's the kiss of death, Robber. Build another site starting now.
I guess the next thing to do then is to ban slurp - if they aren't gonna include the site, then they shouldnt chew up bandwidth.
I know this has been said many times before but it would be nice to have an InktomiGuy here in WW to answer some of our questions, and put some of our fears to rest!
I wouldn't bother banning slurp. You'll still get a few stray visitors on your PFI pages until they expire. Slurp probably will quit visiting all your sacked pages, including the PFI pages once they expire.
Nah, we havent got any PFI pages - that must be really frustrating for those of you that have paid and tried to observe the guidelines and still got banned.
Yeah, Robber, it's frustrating, but getting trashed makes you smarter. A few hundred bucks for a dose of smart pills isn't a bad deal.
| This 39 message thread spans 2 pages: 39 (  2 ) > > |