Searching through the forums here I see that there are a lot of generic scripts or custom solutions being used to block 'rogue bots' from entering sites. We have our own custom solution that does this and, for the most part, it works ok. However, it's becomming very onerous to manage and is far from perfect.
We have trialed a 'web application firewall' with our hosting service and it has limitations. There is virtually no reporting (ie, we have to take their word that they are blocking bad bots and letting all the good guys through) and we have no proof that it actually works.
Also, we have looked at other services like [
siteblackbox.com...] which seems to be ok on the surface of things.
If anyone has any experience or knows of any alternative solutions, either SAS or appliance it would be very interesting hear your thoughts.
Thanks, Karl
NB: for what it's worth we regularly serve a few million page views/day to humans and can serve upwards of 10 million page views/day to bots (both search engines and other).
