Welcome to WebmasterWorld Guest from 54.234.38.8

Forum Moderators: mack

Message Too Old, No Replies

Does msnbot definitely obey crawl-delay?

hit for 700MB in 4 days

     
2:18 pm on Aug 24, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Mar 10, 2003
posts:273
votes: 0


Does msnbot definitely obey the robots.txt crawl-delay?

I've just been hit for 700MB in 4 days, and would like to know what to expect from now on.

I know others have been hit even harder - Surely the default crawl rate should be stepped down a bit without special instructions? It's not the only spider around, after all!

2:43 pm on Aug 24, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 22, 2002
posts:1001
votes: 0


Are you sure you used the correct syntax for crawler-delay?

I was being hit pretty hard on a few of my sites - put up the command and things improved.

3:12 pm on Aug 24, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Mar 10, 2003
posts:273
votes: 0


No, that was before I put in the command. I was looking for confirmation that things will definitely improve on the next visit, so thanks for your reply.