incrediBILL - 10:16 pm on Jan 25, 2013 (gmt 0)
any thing in particular you'd recommend to check for within the server headers, and do/have you ban based on something within the headers?
Browsers always send the same headers, over and over and over. If one is missing, like Accept Language, it's a bot. There are some fake headers out there that identify some bots, but mostly if it's a real browser it always does the same thing.
Just beware that some hosts modify some Apache header fields so for instance HTTP_CONNECTION which is typically keep-alive is set to close, OOPS! I used to use that as a signal but thanks to some hosts it's not reliable.
Just like everything else, I whitelist headers. I know what browsers do so I whitelist that behavior and if it's something less than all the headers expected I know it's a bot. If it's something extra, it might be a bot but it has to be evaluated.