Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Yahoo! Slurp
dstiles - 9:50 pm on Sep 11, 2011 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/4360952.htm
I have all of the YST IPs blocked. I only allow crawls from crawl IPs.
98.137.72/24 is pretty much a slurp range but without the crawl rDNS. I wonder why they do this - perhaps it's an excuse to get images?
Brought to you by WebmasterWorld: http://www.webmasterworld.com