Asking about why MSNbot is hitting my server very hard
stinkfoot
2:24 pm on Nov 22, 2005 (gmt 0)
Have seen a massive increase in hits from MSNbot from about 30 to 100 a day to what is now 1000+ hits per day. Anyone esle seeing this?
If not anyone else know who I ask to calm it down?
jdMorgan
2:52 pm on Nov 22, 2005 (gmt 0)
According to the MSN webmaster help page [search.msn.com], you should be able to handle this in robots.txt:
# Limit msnbot to 100 requests per day (86400 seconds) User-agent: msnbot Crawl-delay: 864
Jim
Lord Majestic
2:56 pm on Nov 22, 2005 (gmt 0)
I am just wondering here - do you get hammered by bot from the same IP, or is it multiple MSN bots coming from different IPs?
stinkfoot
12:40 am on Nov 23, 2005 (gmt 0)
Thanks jdMorgan
Not sure about the IP addresses tbh LordM
Zaphod Beeblebrox
9:33 am on Nov 24, 2005 (gmt 0)
I have the same thing going on. Could you check the IP numbers and verify that the ones that do have rDNS specified create roughly the same traffic as before, and that the ones without rDNS are responsible for the extra traffic?