Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Yahoo! Slurp


dstiles - 8:48 pm on Sep 16, 2011 (gmt 0)


Haven't seen anything like this. I have about 300 hits by Slurp/3.0 out of 62,000-ish so far this month (checked only for this month!).

I did get 13 hits from the range 98.137.72/24 this month from Slurp/3.0 but those are killed as being non-crawl IPs.

On the other hand, I did get about 50 hits from googlebot within a couple of minutes today, all on the same (permitted) page. That was very odd.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4360952.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com