Forum Moderators: mack

Message Too Old, No Replies

MSN spidering too hard

         

bazza9

10:43 am on May 3, 2005 (gmt 0)

10+ Year Member



MSN seeems to have been picking up a page every 3 seconds for most of the morning.

I've tried to deny the IP address access and have added a line to my robots.txt to encourage MSN to only pick up a page every 120 seconds from now on.

Are there any other precautions that I can take, as my web host took my site down at one point this morning because they became worried about the load on the server

imstillatwork

6:15 pm on May 3, 2005 (gmt 0)

10+ Year Member



If your host can't handle a page every 3 seconds.... time to move on...

How long has msn been doing this? generally a search engine spidering your site heavily is GOOD thing...

Receptional

3:48 pm on May 5, 2005 (gmt 0)



Presumably the robots.txt is using the protocol as outlined at:

[webmasterworld.com...]

If not, then that should fix it.