Another strange entry

"slurp"

[dcrombie]

66.196.67.101 - - [14/Apr/2004:04:23:59] "GET /dir/file.pdf HTTP/1.0" 200 29537 "-" "slurp"
66.196.67.101 - - [14/Apr/2004:04:04:46] "GET /page.html HTTP/1.0" 200 7143 "-" "slurp"

name = zj1000.inktomisearch.com.

[softbug]

virus?~

[tedster]

The IP address and the domain "inktomisearch.com" both do belong to Yahoo! -- but I can't find them in any of my server logs so far.

Did the bot ask for robots.txt?

[dcrombie]

No robots.txt - but I've had six similar page requests today from the same IP and agent. Slurp is crawling all our sites and (as usual) requesting the robots.txt file like crazy - but not from that IP.

Side note: I emailed them a while ago to ask if it was really necessary to download the robots.txt file for a site 50-60 times in one day. Their response was that it's necessary "because the Slurp crawler is 'distributed'". I almost blocked them for incompetence but that would penalise our sites ;)