| Welcome to WebmasterWorld Guest from 22.214.171.124 |
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
|Subscribe to WebmasterWorld|
|Does msnbot definitely obey crawl-delay?|
hit for 700MB in 4 days
| 2:18 pm on Aug 24, 2004 (gmt 0)|
Does msnbot definitely obey the robots.txt crawl-delay?
I've just been hit for 700MB in 4 days, and would like to know what to expect from now on.
I know others have been hit even harder - Surely the default crawl rate should be stepped down a bit without special instructions? It's not the only spider around, after all!
| 2:43 pm on Aug 24, 2004 (gmt 0)|
Are you sure you used the correct syntax for crawler-delay?
I was being hit pretty hard on a few of my sites - put up the command and things improved.
| 3:12 pm on Aug 24, 2004 (gmt 0)|
No, that was before I put in the command. I was looking for confirmation that things will definitely improve on the next visit, so thanks for your reply.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
© Webmaster World 1996-2013 all rights reserved