AlexK - 8:16 pm on Nov 23, 2005 (gmt 0)

incrediBILL:

A simple solution I found to slowing and stopping unauthorized bots ... start serving up error pages unless it's an authorized bot

My site has similar experiences to your own, and to this site (though at a much, much lower level).

I use a similar automated solution [webmasterworld.com] to stop them, which works fine. The difference is, I do not distinguish between "good" bots and "bad" bots. If the bad bots behave well they get through, and if the good bots behave badly they do not. What other criterion should we have? Any other attitude says that we are harlots (they wave enough money and we let them in - definition of a harlot).