How to avoid counting bot clicks when tracking clicks on ads?

Set a cookie, since bots usually can't accept them?

[MichaelBluejay]

I'm manually tracking clicks for the banner advertisers on our site. Right now the banners link to my tracking cgi script which records the click data into a database (ip, timestamp, referring page, target page, which ad, which ad position), then redirects the user to the advertiser's site.

My tracking script automatically filters out a dozen or so known bots, but as you know lots of bots pose as IE by faking the user agent. To get around this I manually go through the click database once a week and remove clicks that look like bot clicks (e.g., something that clicked every single banner in every single position in the same minute, and which didn't click any of the few banners we have which are generated by Javascript).

This works okay but its tedious, and there are many times I'm not sure whether clicks are bots or not. So I was trying to think of another way to do it. I figured that I could set a cookie and then try to read it, and if I failed to read it then assume the agent is a bot and don't record the click. I wouldn't get to count real people who have turned off cookies, but I imagine they're less than 5% of users, probably less than 1%.

So my question is: Does this seem like a good idea? Seems like it to me, but I'm new at this and could be missing something. Is there a better way to exclude bot clicks when tracking ad campaigns?

Incidentally, the solution does have to be home-grown -- I need the data stored in my own database so I can easily customize the reports that advertisers see and do other things with that data with scripts. So I can't use any off-the-shelf software.

Thanks for your help.

[Reid]

A browser would load every image but a bot wouldn't load any. Put some code that will compare an image being called by a browser during a pageview that a bot just wouldn't do. A 304 would definitely be a browser that has images cached. So if it retuns a 200 and the image is not called - likely a bot (or an old browser but very low % on that).
I'm not into SSI at all but it seems like a possible approach.

[MichaelBluejay]

That's an interesting idea. But I'm afraid it's also over my head. I'd have no idea how to implement that.

[Lord Majestic]

I'd have no idea how to implement that.

Have 2 scripts:

1) Ad script will record your normal things (IP, time etc) and, if its first request from that IP/useragent in the last X hours then mark it as BOT, if its second or more requests and it is still marked as BOT then don't forward click

2) Another script will be included as 1x1 pixel image written into code of the page using JavaScript write function -- this script would execute after actual click occurred and it should unmark previous request that was marked as BOT.

[MichaelBluejay]

I'm afraid this still doesn't make any sense to me. What about my original idea of only counting clicks if a cookie was accepted. Would that work? I would be able to count users who have cookies on but Javascript off. With the method just suggested I think I wouldn't get good data if users had Javascript off.

[Lord Majestic]

Cookie based check should work with most bots, however be aware that its much easier to fake cookies support than JavaScript support.

[Reid]

It really depends on your page design. Lots of people have cookies and or Java disabled.
you could do

<IMG src="php or CGI script" height= "1" width="1">

It's not really an image and will only make a 1x1 pixel dot on the page. A bot won't even fetch the script and it's SSI so it doesn't matter if the browser has cookies or not. You could even send a 1x1 image from the script.

Now the question remaining - what to do with the script. If the script is called for that pageview the clicks are valid. Here is where it gets beyond me but if you go into the CGI or PHP forum (whatever your server supports) I'm pretty sure they could help you with it. Or you could match referers for the page with referers for the script, or match referer for the script with referer for the link, if it has both it's a click.

[Reid]

I just thought too - if your already tracking clicks on the banner then you just need to track requests for the banner image as well. If an IP requests the banner image then it's flagged as valid. But if it requests the link but has not even 'seen' the banner then it's a bot.

[MichaelBluejay]

I just thought too - if your already tracking clicks on the banner then you just need to track requests for the banner image as well. If an IP requests the banner image then it's flagged as valid.

Here again, I have no idea how to test for this.

I think I'm just going to use my original cookie-writing idea.