keyplyr - 9:02 am on Sep 17, 2011 (gmt 0)

keyplyr, did that Slurp/3.0 read robots.txt?

No request for robots.txt, just hit a popular page and took all the files, then another page etc... like a browser would.

With further examination it did *not* take the entire site, just looked that way at first glance :)