Forum Moderators: goodroi

Message Too Old, No Replies

Bot blocking versus IP throttling for bad bots

Dealing with rippers and their ilk

         

Webwork

4:10 am on Mar 3, 2005 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



After I set up a robots.txt file as described in other threads around here what, if anything, can one do to tame 'bad bots'?

What about allowing deep/rapid crawling only by bots from the known IP ranges of the major SEs? For any other IP that is attempting a rapid and deep crawl launch an error message?

Has it been tried? Failed or worked? Tried with XYZ consequences?