How to clean up your traffic. Avoiding bad bots, click bombing, etc.

[MrSavage]

This is my crossroads post. One path is failure and the other is unknown. I really only have one way to go on this.

I've been reading the monster thread about click bombing, however, it's a year old at this point and no replies are allowed. However since it's a year old, I'm sure a lot has changed. That's largely why I'm posting this.

So my story is this. I've seen drastic earnings reductions. I've also seen over the past while, situations where I've seen a massive earnings total from an ad unit, only to check back in 20 minutes and see that massive earnings gone. This is obviously a very troublesome situation.

I've contacted Adsense support and as expected, they've mentioned invalid click activity. Once again that phrase of quality of traffic comes up.

So for me, the situation is this. The only way I can logically determince my future with Adsense is to remove the issue around my site traffic. I can't blame Adsense at this point. It's all about what I'm going to do now moving forward. I don't believe my stats these days and picking off single IP addresses based on Awstats isn't doing nearly enough.

Netmeg on another thread mentioned the value of Google Analytics regarding catching bot activity. That's a step one in this process. I can honestly say based on what I've been reading, this is a daunting task. Blocking IP address can be problematic when you start blocking unintended targets. Messing with .htaccess and wide IP blocking is also risky. Using bot traps is no exactly a simplistic venture.

So this is hopefully going to be a resource. I will post my tips as I go along. I'm not looking forward to this because for the most part, a lot of this is a clear as mud. I will also state here, that it's completely baffling to me at the lack of tutorials, guides, easy to understand anything regarding bots and bad traffic. If it's so critical to Adsense, and there are so many published, then why the F is there next to no #*$! resources? Sure they may be out there, but if I'm here asking for a hand, I'm asking because there is minimal guides out there. I even found the Google help pages all but futile which to me, at this time, is a bit baffling. Clean up traffic? How so?

I'm all ears. Perhaps people want to keep some of their tactics to themselves, that's fine. My first move is Analytics and using that block of IP's from the click bombing thread from yesteryear.

Sorry to be long winded. I'm just puzzled and a bit baffled why cleaning up traffic is not a front and center discussion when right now, it appears to be EVERYTHING.

[netmeg]

We do have a forum for that, but in my opinion it's easy to miss because of how it's named

[webmasterworld.com...]

It started out more about search engine spiders, but nowadays it's probably more about bots (good and evil)

At any rate, it can be a monumental task, so good luck with it. One way to start might be taking a look at logfiles and/or traffic patterns over the past years, and see if you can identify your good/normal/regular traffic, and then start looking for anomalies.

Look for things like, country of origin, time on site, bounce rate (to some degree), pageviews per user session, acquisition source - basically you're looking for signals that point to real live human behavior. Then take a look at the outliers.

Analytics won't give you IP numbers (they can't for privacy reasons) but it *will* give you ISPs, and I have identified a few that seem to send a whole lot of bot traffic (including Amazon) For IP numbers, you might want something like Piwik or even StatCounter if you check it regularly enough can give you some idea of when anomalies occur.

[Mike343]

You know I was told this same thing last month, one day the CTR just nose dived and the best answer they could give me was "potentially invalid traffic from foreign countries but I do monitor my traffic and it has not changed much at all in the last 2 years concerning countries and traffic sources (70% organic, 30% direct)

So I am thinking it is probably a very overly aggressive algorithm they implemented to deal with attacks which they won't even tell us that they did implement it.

[Mike343]

Also without making changes to the ad code you could wrap it in a div then wire up a javascript function which fires off an ajax event recording the ip address that clicked within that area as well as the date and time.

You will have to take into account that the div might contain an iframe within it and monitor if anything was clicked within that.

[not2easy]

I wish I could tell you that an automated process has been developed, but, no. The best way is to monitor your access logs in whatever way is convenient to you. They can be viewed as an EXcel sheet, an OpenOffice spreadsheet, a text file or using one of your host's tools. It is getting worse, not better. I have been blocking for about two years and never see any "clawbacks" for AdSense so it definitely helps in that respect.

[netmeg]

So I am thinking it is probably a very overly aggressive algorithm they implemented to deal with attacks which they won't even tell us that they did implement it.

I believe there's some of that going on as well, because if I were Google, that's probably what I would do. If the advertisers start losing faith in the network, it all collapses. If the publishers start losing faith - well, there's always gonna to be more publishers. Plus Google still has all the search network ads.

But no, they will never tell us that, and I have no proof that it's happening. Just a hunch.

[Mike343]

That is true that they will always have publishers but to what extent can they afford to lose publishers. The content network is a fairly large part of it.

As for not telling us anything I don't think it would hurt to be honest as security by obscurity is not security. Just say yes we implemented a new algorithm for invalid traffic or no we didn't, I don't want to know the specifications of it but like you said G will probably never speak a peep about this and I know from experience contracting on other projects within G not related to the advertising sector at all.

Back on topic, look into your foreign countries and see which ones have provided more page views then the others. From there I would filter those countries out and see what happens.

I did this using dfp and noticed an increase in cpc and less clawback but DFP is less then ideal with adsense at the moment.

[netmeg]

Just say yes we implemented a new algorithm for invalid traffic or no we didn't

Well last year they did purchase a company with some amount of expertise in detecting invalid activity. I guess that's as close as they come to disclosure.

[MrSavage]

Based on the rocky road as of late (more than just a couple months), I don't see how I can actually determine that the issue is Adsense and not in what I'm doing or not doing. I've been blaming Adsense directly and that's gotten me nowhere. Dealing with my traffic is something I can deal with and it's going to determine whether my time is worth this investment in running websites. If I combat the crud visiting my sites AND I continue to suck, then it's time to close out a lot of ventures and ideas.

I appreciate the link to the other webmaster forum. As I've found thus far, this is humbling in the sense that it's complicated right now to figure out. I hope that at the end of this I can simplify some instructions on how to clean up your website traffic. I think a lot of this gets bogged down amongst those who already understand what to do, and thus, the newbie such as myself is left scratching their head and ultimately give up.

[nmjudy]

I'm confused. Tell me what I'm not seeing...
I read through the thread that Netmeg referenced in determining bad bots using Google Analytics and Piwik or Statcounter.

If I look at Google Analytics under "Behavior/Adsense/Adsense Pages" - I can see how much money each page has earned according to Analytics. Analytics says it is showing 100% of the earnings. When I compare Analytic Adsense earnings to what the Adsense Performance report is showing, the Analytic Adsense earnings are less than half of what Adsense itself is showing. Yet I'm being paid the higher amount less several dollars.

Example: February 2015 earnings for the month, "Google Analytics Adsense Pages" show less than half the amount that Google Adsense Performance report states for the month. Yet, the amount I will be paid is $15 less than what the Adsense Performance report states (with no reason why or notation of any kind of invalid click activity). The amount being paid is still almost twice as much as what Google Analytics Adsense Pages is showing.

For the past year, I've eliminated showing ads to all direct traffic. This leaves a HUGE amount of ad inventory unfilled. Prior to making this change, I never saw an adjustment in Adsense earnings until it was posted at the end of the month (my performance reports never showed an adjustment). One month there was a $2000 difference in what was showing and what I got paid. No explanation. Occasionally, I'd see "invalid traffic" adjustments on the transaction report. But they've always been under a dollar.

I wonder if my issue is something different than click bombing. Where/when are you seeing amounts being adjusted?

[MrSavage]

So far, this is rough slogging. It was one year ago that the click bot thread was last posted in. Would it be advisable at least to implement those ip address blocks? I'm just wondering if those people who added them at that time, if they would still be relevant. I guess boiled down, I'm wondering if 6 months to a year information is largely not useful in adding now. I'm sure it's not possible to give a cut and dry on it, but overall, these targets change in weeks? I know it's whack a mole, but having some idea how far back I should go might help a bit. Perhaps there is no answer to that question?

[MrSavage]

Update: I documented a click bombing attack in February. I noted the amount, the time and date and the time when the amount was removed (about 20 minutes later). At this point, I'm sorting through raw access logs and look at activity prior to that noted time when I saw the attack. A big head scratched at this point is seeing in raw logs, IP addresses that are highly suspicious, yet in awstats, I can't find those IP addresses listed in my same logs. I know that when I sent an email to Google about this activity, I could have submitted IP addresses to them. At the time I didn't have any. The point is, this is an involved process. However, what's more important than tracking down a click bombing attack?

[MrSavage]

@netmeg, I realize the original post from a year ago cannot be revived, but could I ask something? When you narrowed down the offending IP addresses, was this a situation in your logs where it was a couple odd entries in a short time frame or was it one IP will a massive amount of entries in a short time? I believe you utilized Analytics which helped you, but I don't have that for this project.

In general, when I'm viewing my raw access logs, there are those ips that have chunks of entries at one time vs. ips that have one or only a few entries at one time. In your experience are they equally suspicious on that criteria alone or is the large chunck of entries from a single IP more indicative of "normal" behavior? Thank you.

[netmeg]

I wish I could help you more, but I'm really slammed for time these days.

When I was getting tons of clawbacks, I tried to catch periods where my AdSense clicks were outside normal range, and tried to look at Analytics and/or log files before the clawback occurred, so I could take a "snapshot" of the data, and see if those clicks stuck around.

Without any analytics in place, you're going to have a real hard time trying to reverse engineer it from old logfiles. You might be better served to put up Analytics for a month and then see what you can get out of it.

But for the click attack you mentioned - the fact that they show up in your raw logs but NOT in your awstats is probably a pretty good indicator that was bot activity, as I believe most awstat configurations filter bots by default. Now, whether or not those are ad clicking bots, I dunno. But it's a pretty good bet they're bots of some kind. You might try plugging a few of them into a tool like this [iplocation.net...] and see if any patterns emerge.

[MrSavage]

Thank you very much. I can appreciate what you said about time. There is never enough.

That's a very interesting tidbit regarding awstats filtering out bots. That's a key piece of helpful info.

No question I would rather be writing or creating content, but I've come to realize this is likely as important, if not more important. I'm a bit frustrated that I don't have the tools/data that I likely need to get the buggers who clickbombed me. Lesson learned.