Forum Moderators: open
It has not requested robots.txt this access_log file session week. Then again, maybe it only does that request periodically.
Fourteen files in two seconds seems rather hungry.
[edited by: martinibuster at 6:01 pm (utc) on Jan. 16, 2005]
[edit reason] correction. [/edit]
It's a little frustrating as the site:www.mysite.com on Y is yielding X pages, yet slurp is gobbling 3X pages per day.
Hopefully something good will come out of this besides my bandwidth spike (which isn't that good).