Currently it has a list of "bad bots" and a simple filter to make sure the user-agent looks "normal", however a thought occurred to me today and I was curious if anyone else knew a little more on the subject;
Given that most of these programs heavily market how fast they go it would stand to reason they'd want to keep overhead to a minimum, therefore keeping their requests as short as possible.
Does anyone knows how much effort these programs put into faking i.e. do they just simply modify the user-agent or do they go the whole hog and supply the correct extended headers such as "Accept:" & "Accept-Content:" etc.
If this *was* the case it could potentially mean you could identify bots pretending to be regular browsers by their lack of headers.
Any thoughts?
