- - Search Engines
- -- Sitemaps, Meta Data, and robots.txt
- ---- In the era of increasing numbers of bad bots is robots.txt irrelevant?
incrediBILL - 9:21 pm on Feb 21, 2006 (gmt 0)
|but I can't see how any of this whitelisting stuff can work |
I bounce thousands of page requests a day from whitelisting alone as all the random bot names that come along just bounce off the wall, they get nothing.
|but what about those which use regular browser strings ... you aren't going to stop a bot with Internet Explorer's User-Agent string that makes only 100 requests in a day |
Server side script analyzes their behavior and challenges or bans them as well, so I'm stopping them and people using AlexKs script might be as well.
It takes multiple techniques to squish the nonsense, but it can be squished for the most part.
Thread source: http://www.webmasterworld.com/robots_txt/871.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com