Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Yahoo! Slurp

dstiles - 9:50 pm on Sep 11, 2011 (gmt 0)


I have all of the YST IPs blocked. I only allow crawls from crawl IPs.

98.137.72/24 is pretty much a slurp range but without the crawl rDNS. I wonder why they do this - perhaps it's an excuse to get images?

Thread source:: http://www.webmasterworld.com/search_engine_spiders/4360952.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com