Forum Moderators: goodroi

Message Too Old, No Replies

Define allowable spider times

MSN is going nutz!

         

Reno

8:51 pm on Dec 29, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I have a relatively small website -- under 15 MB in size. The MSN bot comes every day and hits it hundreds of times -- today alone msnbot accounted for 43 MB of bandwidth. Multiply that over the course of a month and you can see the problem.

Is there a way for me to utilize robots.txt to limit the spidering of the site by msnbot to only once or twice per week? For example, it would be allowed on any Tuesday and/or any Friday?

At this rate I will have to buy more bandwidth just to account for what this one bot is using, which seems silly. Any advice is appreciated....

..........................

Reno

1:44 pm on Jan 4, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Success! For the last 3 days the msnbot has averaged only 5 MB per visit -- down from a consistent average of 40+ MB/day prior to the modification of robots.txt.

Thanks again for all the very helpful advice...

.......................................

This 31 message thread spans 2 pages: 31