I have a relatively new site (started March 2001, no real links until March 2002). I never paid inktomi a cent, and got essentially zero hits from MSN.
In August I noted a substantial number of MSN hits (about 10% of my SE traffic), and at that time others started posting that it seemed that Inktomi had started spidering much of the whole web, whether the sites had paid or not.
I know nothing about the status of BOW, but clearly they are spidering many sites for free.
Mohammed, I could relate to that experience. There was more spidering and hits from MSN .. but in the last month or so, all those hits disappeared virtually ... making me wonder if Inktomi dropped the ball on BOW (free inclusions)
I just used PositionTech to check, and my freebie pages are still in Ink. I don't know if they're "Best of Web" -grade stuff, but they're still there. They appeared shortly after the sites were listed in the ODP, and I suspect that is why they are still listed in Ink.
Interestingly, only the home pages of the sites are in ODP, but all pages are included in Ink, so the sites were fully spidered as a result of ODP inclusion - that's my theory, anyway.
I'm not sure how Ink works these days, but I'm grateful for the listings!
Inktomi vastly expanded the number of pages within their database on August 8th.
Many of the pages included were automated doorway pages that were cloaked, spanning multiple domains forwarding to common sites or mirror sites.
On popular searches these well "tuned" pages now appear at the top leaving "BOW" pages on following pages.
A. Aggressive formatting becomes a necessity to break above these pages.
B. Reporting the offenders removes them in around 60 days.
Defend, Protect and Police your home turf.
Please report spam when you see it. As minnapple mentioned, we've significantly expanded our database of late.
Secondly, the Best of the Web index has gone away as a separate database. Now that we've rearchitected, we don't need a separate database of frequently-crawled content. Everything is frequently-crawled content :-)
|Everything is frequently-crawled content |
I will agree with that. I have noticed slurp/cat in my logs daily. They have crawled my sites more this month than the past 5-months combined :)
Inktomi, yes it seems Slurp is crawling a lot ... but the site pages fail to stick in the results. Can you tell me what is going on?. I sent you the site as a sticky.