| Welcome to WebmasterWorld Guest from 22.214.171.124 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
|Does msnbot definitely obey crawl-delay?|
hit for 700MB in 4 days
Does msnbot definitely obey the robots.txt crawl-delay?
I've just been hit for 700MB in 4 days, and would like to know what to expect from now on.
I know others have been hit even harder - Surely the default crawl rate should be stepped down a bit without special instructions? It's not the only spider around, after all!
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved