- - Search Engines
- -- Search Engine Spider and User Agent Identification
- ---- Yahoo! Slurp
Pfui - 10:14 pm on Sep 11, 2011 (gmt 0)
Or they're developing the equivalent of a snapshot/Google Web Preview thing? Or they could care less and less about robots.txt? ;)
Note that "Yahoo! Slurp/3.0" was a sure-fire culprit at my end and it crawled from .yst and .crawl servers.
Thread source: http://www.webmasterworld.com/search_engine_spiders/4360952.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com