Slurp looking for non-existant files

[dhatz]

I noticed many 404 errors from Yahoo Slurp crawling an old site, looking for non-existant URIs.

At first I thought it was broken links, but I've checked the site and there don't seem to be any. Also I don't get 404s from anyone else, users of robots crawling the site, APART FROM Inktomi before Yahoo

It seems it is a bug of Slurp (btw the destination pages are redirected via 301 from another URI) or might it be a "test" of Slurp to make sure it's not a fake site that generates seemingly valid pages for any URI? But there are just too many 404s everyday to be just a "test"...

This behaviour has been going on for months (I checked back to Jan-2004), only with Slurp from Inktomi/Yahoo, all other robots never produce 404s

Any ideas?

66.196.90.152 - - [27/Mar/2004:08:18:28 -0700] "GET /example/083.html HTTP/1.0" 404 211 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
66.196.90.50 - - [27/Mar/2004:09:36:49 -0700] "GET /example/884.html HTTP/1.0" 404 212 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
66.196.90.184 - - [27/Mar/2004:09:52:17 -0700] "GET /example/384.html HTTP/1.0" 404 211 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
66.196.90.83 - - [27/Mar/2004:10:22:53 -0700] "GET /example/377.html HTTP/1.0" 404 212 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
66.196.90.64 - - [27/Mar/2004:10:59:56 -0700] "GET /example/1794.html HTTP/1.0" 404 213 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"

[webmktg]

I'm experiencing a similar situation for my websites. I think this is the natural behaviour of Yahoo Slurp.

[8cool]

I have the same problem. Check Yahoo_Mike's Answer for my question [webmasterworld.com...]

But I feel there is something wrong with Slurp

[dhatz]

Thanks for the feedback. As I suspected, it's some sort of "check" by Slurp. Explained by jdMorgan at

[webmasterworld.com ]

But I hope Yahoo will limit the number of 404's per day, and to no more than say 1-2% of REAL requests. Over the last few days, for the abovementioned site, it was over 11%, ie there were

183 404s
1663 200s

In my case it was a false alarm and I wasted a couple of hours doing integrity checks for broken internal links and searching via the SEs for potential broken backlinks, trying to fix a non-existant problem ...