-- Search Engine Spider and User Agent Identification
---- Quick primer on identifying bot activity.
Ocean10000 - 12:48 pm on Apr 4, 2008 (gmt 0)
Thanks, I had come to conclusion today that I would run a risk relying on public IP lists, and had found a few pages about spider traps and such that I thought may be a better solution. Also as you say to keep any kind of IP list updated could be difficult.
Is it these kind of spider traps that you use? I saw a few such as a little test to see if the bot followed robots.txt rules, that sounded like a good idea, do you do stuff like that?
Yes I use a custom written module for Asp.Net which allows me to filter out the unwanted traffic, by following the steps I outlined in my posts here and then some.