Welcome to WebmasterWorld Guest from 54.225.20.73

Forum Moderators: mack

Message Too Old, No Replies

Does msnbot definitely obey crawl-delay?

hit for 700MB in 4 days

     
2:18 pm on Aug 24, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Mar 10, 2003
posts:273
votes: 0


Does msnbot definitely obey the robots.txt crawl-delay?

I've just been hit for 700MB in 4 days, and would like to know what to expect from now on.

I know others have been hit even harder - Surely the default crawl rate should be stepped down a bit without special instructions? It's not the only spider around, after all!

2:43 pm on Aug 24, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 22, 2002
posts:1001
votes: 0


Are you sure you used the correct syntax for crawler-delay?

I was being hit pretty hard on a few of my sites - put up the command and things improved.

3:12 pm on Aug 24, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Mar 10, 2003
posts:273
votes: 0


No, that was before I put in the command. I was looking for confirmation that things will definitely improve on the next visit, so thanks for your reply.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members