- Search Engines
-- Search Engine Spider and User Agent Identification
---- Yahoo! Slurp
- 9:02 am on Sep 17, 2011
keyplyr, did that Slurp/3.0 read robots.txt?
No request for robots.txt, just hit a popular page and took all the files, then another page etc... like a browser would.
With further examination it did *not* take the entire site, just looked that way at first glance :)
Brought to you by