Welcome to WebmasterWorld Guest from 23.22.46.195

Forum Moderators: mack

Does msnbot definitely obey crawl-delay?

hit for 700MB in 4 days

   
2:18 pm on Aug 24, 2004 (gmt 0)

10+ Year Member



Does msnbot definitely obey the robots.txt crawl-delay?

I've just been hit for 700MB in 4 days, and would like to know what to expect from now on.

I know others have been hit even harder - Surely the default crawl rate should be stepped down a bit without special instructions? It's not the only spider around, after all!

2:43 pm on Aug 24, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Are you sure you used the correct syntax for crawler-delay?

I was being hit pretty hard on a few of my sites - put up the command and things improved.

3:12 pm on Aug 24, 2004 (gmt 0)

10+ Year Member



No, that was before I put in the command. I was looking for confirmation that things will definitely improve on the next visit, so thanks for your reply.
 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month