homepage Welcome to WebmasterWorld Guest from 54.81.170.186
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Home / Forums Index / Microsoft / Bing Search Engine News
Forum Library, Charter, Moderators: mack

Bing Search Engine News Forum

    
Does msnbot definitely obey crawl-delay?
hit for 700MB in 4 days
Just Guessing




msg:1537420
 2:18 pm on Aug 24, 2004 (gmt 0)

Does msnbot definitely obey the robots.txt crawl-delay?

I've just been hit for 700MB in 4 days, and would like to know what to expect from now on.

I know others have been hit even harder - Surely the default crawl rate should be stepped down a bit without special instructions? It's not the only spider around, after all!

 

Sanenet




msg:1537421
 2:43 pm on Aug 24, 2004 (gmt 0)

Are you sure you used the correct syntax for crawler-delay?

I was being hit pretty hard on a few of my sites - put up the command and things improved.

Just Guessing




msg:1537422
 3:12 pm on Aug 24, 2004 (gmt 0)

No, that was before I put in the command. I was looking for confirmation that things will definitely improve on the next visit, so thanks for your reply.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Microsoft / Bing Search Engine News
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved