Welcome to WebmasterWorld Guest from 54.205.106.138

Forum Moderators: coopster & jatar k

Message Too Old, No Replies

How to accurately record # of clicks on button in db, minus crawlers?

Crawlers/bots are inflating my click count in database, how to fix?

     
5:00 pm on Oct 11, 2010 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 23, 2007
posts: 48
votes: 0


I have a "sign up" button that redirects users to another website (using javascript onclick/popwin). Just prior to redirection (header: refresh), I insert into my own database so I can count the # of clicks on the "sign up" button. However, crawlers and bots are inflating my click count because they are "clicking" on (or in other words, loading the URL of) my "sign up" button. Is there any way I can make my click count more accurate?

Thank!
5:12 pm on Oct 11, 2010 (gmt 0)

Moderator from US 

WebmasterWorld Administrator lifeinasia is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 10, 2005
posts:5646
votes: 66


Filter out IPs and useragent strings of known bots. Add to the list as new bots appear and/or you see them in your logs.
5:38 pm on Oct 11, 2010 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 23, 2007
posts: 48
votes: 0


Is there a more efficient way? Adding to the list would entail sifting through IP addresses and logs every single day. And then, I'd also have to delete from my database to get an accurate count. If possible, I'd rather put some tests in place prior to inserting into my database, but not sure how to test it?

Thanks.
6:03 pm on Oct 11, 2010 (gmt 0)

Moderator from US 

WebmasterWorld Administrator lifeinasia is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 10, 2005
posts:5646
votes: 66


You basically have 2 choices:
1) Filter at input (logging), or
2) Filter at output (running reports)

Option 1 means that you have to do the checking every time before you log a click. Option 2 means you only check once.

Adding to the list would entail sifting through IP addresses and logs every single day.

That's pretty much the only way to keep updated.

not sure how to test it

You have a table of known bots/spiders. You compare the IP and user agent of the click with what's in the table. If there's a match, it's a bot.

You could use JavaScript to open another window, log the click, then do a self.close(). That would eliminate logging of clicks for bots that can't process JavaScript.

But then again, if the bots are firing the onclick event... What happens if you have JavaScript disabled and click on the link?
1:39 am on Oct 14, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 31, 2005
posts:1108
votes: 0


For the well behaved bots/spiders you could add the URL of your sign up button to your disallow in robots.txt

For the badly behaved bots you will need bot traps & banning.

Why not write the URL to the page using JavaScript? Very few bots if any run JavaScript. (Making sure that it doesn't look like a URL in the JavaScript).
4:13 pm on Oct 14, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member rocknbil is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Nov 28, 2004
posts:7999
votes: 0


Is there a more efficient way?


I don't know about efficient, but

I have a "sign up" button that redirects users to another website (using javascript onclick/popwin).


Since you ae using Javascript, and only because you are using Javascript, there may be a workaround.

if (document.getElementById('some-id-element)) {
// Record the visit and do the redirect:
window.location='http://www.example.com';
}

The logic is, only actual browsers would execute the Javascript, eliminating crawlers. The " if document.getElementById" condition will only return true for modern browsers.

The downside is it would also eliminate most mobile browsers.