Welcome to WebmasterWorld Guest from 188.8.131.52
Forum Moderators: open
1. Googlebot 41,735
3. TurnitinBot 7,058
4. Openfind data gatherer, Openbot 493
5. FAST-WebCrawler 180
6. Mozilla/5.0 (Slurp/cat; firstname.lastname@example.org; [inktomi.com...] 166
7. Mozilla/5.0 (Slurp/si; email@example.com; [inktomi.com...] 37
8. scooter 22
9. http: 11
10. BaiDuSpider 11
11. Mozilla/3.0 (Slurp/si; firstname.lastname@example.org; [inktomi.com...] 9
12. Mozilla/3.0 (Slurp.so/Goo; email@example.com; [inktomi.com...] 7
13. Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; MSIECrawler) 4
14. URL Spider Pro 3
15. Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MSIECrawler) 2
16. Steeler 2
17. ExactSeek Crawler 2
18. Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; YComp 184.108.40.206; MSIECrawler) 2
19. Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Q312461; Hotbar 220.127.116.11; MSIECrawler) 1
20. Robot: NutchCrawler, Owner: firstname.lastname@example.org 1
It just seems to check a few pages and that is it? What could the problem be? And is this a poor crawl of search engines for a month? Does MSN and Altavista crawl once a month? I don't see a lot of the big engines every month should I?
I've been watching Ink closely since early September and have seen similar dismal spidering activity. At the same time, I had one free inclusion site and one paid inclusion site dumped ("dumped" means the free sites pages cannot be found at all in their index and PFI pages are absolutely buried at the bottom of the barrel).
I also have a couple new sites put on the Web in September and Google, Alta Vista, Fast and others have picked them up but not Inktomi.
It looks to me like they're unsuccessfully fighting spam for their lives (because the PFI site they dumped had been in their index for over a year, and is not spam as far as I and all the other search engines are concerned) so they are having to crank up their spam cleansing so radically as to cut off their noses to spite their face. Still, it's easy to find tons of spam in their index.
It could also be that they can't afford the cost of spidering the entire web anymore.
And maybe they can no longer afford the cost of enough skilled technical manpower to maintain their systems and algorithms properly. Being from the programming world myself, I know when the chips are down one of the first things management wants to do is cut expensive technical resources and stop thoroughly testing code before it goes out the door, usually with disasterous consequences.
My window for analysis is very small and based on a few small websites of less than 50 pages, but I'm trying to contribute my best two cents worth toward understanding the problem.
What's Inktomi "guy's" take on it all? Some real facts from the expert would help dispell any erroneous conclusions based on limited information.
Had a similar problem with slurp not spidering more than my index page. I included <META NAME="ROBOTS" CONTENT="index,follow"> and now it also spiders the other pages.
Only problem remaining is that it comes once per day, picks just 1 page and leaves. Any cure for this?
Now Inktomi only comes around once a week or so, compared to almost their daily visits before. And as always, they take in the index page and leave.
I have pretty much given up on getting into Ink, since the free route doesn't seem to work and I have a 1500 page hobby web site that I can hardly afford to pay for inclusion.
At least AV and FAST have gotten at least part of my site over the past month. Still no comparison to Google, though, which faithfully "slurps" in my entire site every month.