Forum Moderators: goodroi

Message Too Old, No Replies

Slowing down MSN bot?

Doubling it's bandwidth usage each month.

         

bouncybunny

6:43 am on Jan 25, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



On one of my client's sites, MSNs bots are doubling the bandwidth they use on a monthly basis. Last month was 1.5GB, this month (so far) it is 3GB.

I wouldn't mind, but it's not a particularly large site.

Is there any way to control this beast?

Thanks.

goodroi

1:10 pm on Jan 25, 2007 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



short answer:
User-agent: msnbot
Crawl-delay: 120

long answer:
[search.msn.com...]

bouncybunny

1:48 pm on Jan 25, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks.

bouncybunny

3:33 am on Feb 3, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Just wanted to say thanks. I did a bit of research and also disallowed the following in robots.txt.

User-agent: msnbot
Crawl-Delay: 120

User-agent: msnbot-news
Crawl-Delay: 120

User-agent: msnbot-media
Crawl-Delay: 120

User-agent: msnbot-products
Crawl-Delay: 120

MSNbot on its own has gone from around 140MB a day to a few hundred K a day. Good riddance. My next step would have been to ban the blasted thing.