| Welcome to WebmasterWorld Guest from 126.96.36.199 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
|Does msnbot definitely obey crawl-delay?|
hit for 700MB in 4 days
| 2:18 pm on Aug 24, 2004 (gmt 0)|
Does msnbot definitely obey the robots.txt crawl-delay?
I've just been hit for 700MB in 4 days, and would like to know what to expect from now on.
I know others have been hit even harder - Surely the default crawl rate should be stepped down a bit without special instructions? It's not the only spider around, after all!
| 2:43 pm on Aug 24, 2004 (gmt 0)|
Are you sure you used the correct syntax for crawler-delay?
I was being hit pretty hard on a few of my sites - put up the command and things improved.
| 3:12 pm on Aug 24, 2004 (gmt 0)|
No, that was before I put in the command. I was looking for confirmation that things will definitely improve on the next visit, so thanks for your reply.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved