Slurp has been the most consistent, thorough bot to visit my site in recent months. It identifies new content quickly, and grabs pages in manageable chunks. Today, all of a sudden, I see a boatload of requests for non-existent pages from Slurp - not all in one visit, but in multiple attempts through the night, all nicely preceded by a robots.txt perusal. Is this documented behavior, or is Slurp regurgitating? I just hope it doesn't mind the large number of 404s my site is serving up...
Rob
