Welcome to WebmasterWorld Guest from

Forum Moderators: mack

Message Too Old, No Replies

Does msnbot definitely obey crawl-delay?

hit for 700MB in 4 days


Just Guessing

2:18 pm on Aug 24, 2004 (gmt 0)

10+ Year Member

Does msnbot definitely obey the robots.txt crawl-delay?

I've just been hit for 700MB in 4 days, and would like to know what to expect from now on.

I know others have been hit even harder - Surely the default crawl rate should be stepped down a bit without special instructions? It's not the only spider around, after all!


2:43 pm on Aug 24, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Are you sure you used the correct syntax for crawler-delay?

I was being hit pretty hard on a few of my sites - put up the command and things improved.

Just Guessing

3:12 pm on Aug 24, 2004 (gmt 0)

10+ Year Member

No, that was before I put in the command. I was looking for confirmation that things will definitely improve on the next visit, so thanks for your reply.

Featured Threads

Hot Threads This Week

Hot Threads This Month