Welcome to WebmasterWorld Guest from 54.147.10.72

Forum Moderators: mack

Message Too Old, No Replies

Does msnbot definitely obey crawl-delay?

hit for 700MB in 4 days

     

Just Guessing

2:18 pm on Aug 24, 2004 (gmt 0)

10+ Year Member



Does msnbot definitely obey the robots.txt crawl-delay?

I've just been hit for 700MB in 4 days, and would like to know what to expect from now on.

I know others have been hit even harder - Surely the default crawl rate should be stepped down a bit without special instructions? It's not the only spider around, after all!

Sanenet

2:43 pm on Aug 24, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Are you sure you used the correct syntax for crawler-delay?

I was being hit pretty hard on a few of my sites - put up the command and things improved.

Just Guessing

3:12 pm on Aug 24, 2004 (gmt 0)

10+ Year Member



No, that was before I put in the command. I was looking for confirmation that things will definitely improve on the next visit, so thanks for your reply.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month