Better Monitoring of Google's Adsense Performance Reports

Ideas for improved analysis of data

[wa desert rat]

For many of us, Adsense seems broken. In fact, arguably, even for Google Adsense seems broken if revenue performance is any indicator (see the 2013 Report to Shareholders here [investor.google.com...] and note the performance of the "Google Member Network Websites" which had negative growth in the first three quarters of 2013).

Add to that some analyses by industry pundits (http://www.siliconvalleywatcher.com/mt/archives/2013/10/analysis_what_future.php) which tracks a continuing decline in both Adsense and Adwords revenue since the introduction of Panda.

More specifically to Adsense publishers is that many of us are seeing clicks taken back in greater numbers than ever before. My main site is an outdoors-oriented forum and scores between 500,000 and 1.5 million on Alexa depending upon the season so I am not "big". But I am seeing clicks scraped back to zero twice in one day. Meanwhile Google gives us vague information about why this is going on.

And I'm not the only one.

One big issue for me is that we operate in what is essentially a vacuum of information about our site's performance as far as Adsense is concerned. We have no data on what ads were clicked, who clicked them, why they were scraped off, or even which specific clicks were scraped off.

With this paucity of data from Google publishers have created a paradigm designed to make them believe that they have control over things. But when it comes to Adsense, I suspect that any control is illusory at best. But I can see the temptation. So we talk about positioning, content, design, performance, hosting, etc.

But at the moment some of us - and perhaps most of us - are faced with issues over which there is zero control.

1. Traffic is not the issue.
2. Ad visibility is not the issue.
3. Ad placement is not the issue.
4. Content is not the issue.

The issue is that even though visitors are seeing ads and clicking on ads we are not getting any real information about why those clicks are not generating earnings.

So is there a way we can track the data we can get and see if we can come up with a way to analyze that data and get some useful information from it? And is that information that we can use to either determine that there is something we can change or if it's a problem on Google's end?

So, to that end I've implemented a routine in which I document the "Performance" report from Google at more regular intervals and paste them into a database-based note-taking tool called "CherryTree" which is available for Linux, Mac and Windows workstations (I use Linux). It is available for free (open source) here: [giuspen.com...]

I suppose I could just jot them down, but where's the fun in that?

About 9 months ago I implemented an analytical tool that is separate from Google called "Piwik". Piwik gives me far more than Google Analytics in greater detail. With Piwik I can determine who exactly is active and what they are looking at in real time! Including IP address, operating system, browser, plug-ins in that broser, whether that visitor is a returning visitor or new visitor, and what mobile device they are using. Piwik is available free (open source) here: [piwik.org...]

Piwik installs into your web page code and CherryTree installs into your workstation. Piwik is robust and scalable to quite large web sites.

What I really want is something that will track clicks in real time. If anyone knows anything that will do that I'd love to hear about it.

[wa desert rat]

@webcentric: Two excellent points.

The first point has to do with the fact that the bots are not web crawling. They receive directions from the C&C servers. They don't just hit sites owned by publishers who are in on the scam either, they are directed to a site by a search engine hosted on the C&C servers. Everything is pre-arranged except where the bot will will browse to in order to make the "click".

The only beneficiary from this scam would be Google (who gets paid for the click) and the advertising agency (who makes a commission from the ad click). They really don't need a web publisher to be complicit in order to run the scam. (Although a web publisher - like Google - would indirectly benefit from this scam as a result of extra clicks.)

Google has already been involved in litigation about this and are probably not eager to have another round with the advertisers.

The bots know the high-bid ads because the ad agency that configured the bots placed the bids. The bots receive a list of ads to click on from the C&C servers that includes ads not in on the scam in order to obfuscate the trail.

And the bots are specifically programmed to appear to be humans web surfing. They even introduced a "dithering" algorithm that makes them appear to move like a person.

These are very difficult bots to track and detect. No simple IP address ban will work - not even an ACL at the router - because the bots are not hosted on servers but on infected PCs. The servers they contact merely act as Command and Control agents.

If you know the IP address of the C&C servers, however, you can stop the click-bots in their tracks by blocking outbound packets to those IPs. I wonder just how many of these are out there. The graphical displays of the click-bot activities in the Berkley "farm" was impressive.

[edited by: wa_desert_rat at 4:38 am (utc) on Mar 5, 2014]

[webcentric]

@netmeg I don't believe the only use for such a discussion is to somehow try to fix google or it would be a pretty acidemic exercise at best. I do think people need to better be able to understand how to use performance reports when it comes to identifying things they may be doing that could be contributing to something they are experiencing...I for one am not experiencing this yoyo like some which leads me in two directions...1. I haven't been targeted by a bot or 2. I'm doing something differently than other people that doesn't lead to certain types of takebacks. Using performace reports, I think it is possible to identify such things as a problem ad unit placement, hightened takebacks related to specific types of devices or countries, etc. It think many people are focused so intently on the bouncing ball that they're not seeing the details and sharing some "general" insights could help us all narrow the problem down a bit. For example, it may be the case that some of these issues may be device specific or related to specific countries or something else identifiable. A little shared brain power might actually uncover some trends non of us expected and I don't think it means sharing specific stats to get it done. It's quite interesting how many people share stats on this site that are against G's terms of service to share so I agree with you on that point. Still, I don't think one needs to share CTR, RPM or even revenue amount to get to the heart of certain issues. If you see a trend in desktop takebacks and someone else can confirm that while both also see less of a problem with mobile devices, that's a clue. I don't know where it leads but maybe it tells you to put your emphasis on mobile for awhile. What do I know what someone can do with a solid, reliable piece of information? All I know is all the complaining is going nowhere but a little cooperative effort could go a long way, if only to just get people to settle down and not make rash decisions based on things that they are experiencing absent any real understanding.

[wa desert rat]

@Webcentric

Netmeg has mentioned the problems of reporting issues which were exacerbated when Google made them more "real time". Just the issues between the Performance Reports and the Analytics reports illustrate some of these issues.

It does seem counter-intuitive that Google would move in a direction to essentially cripple what had been a major revenue source. But G has their own publishing arm which makes a LOT more money than Adsense/Adwords does (especially lately). In fact that group surpassed the "member network" in about 2003 and has never looked back.

Our salvation may be the entrenched bureaucracy that inevitably forms a major part of any large organization. I suspect those folks will fight to retain the member network. In fact, if they can show that we're actively working to help identify the problems they face we might come out looking even better.

[wa desert rat]

I want to also emphasize that our web sites are not "targeted" by a bot. It's the ads they are interested in and since they have a way of determining which ads are displayed on a site and when, that ad is the only criteria.

If an ad the bad guys want to click on is going to appear on your site, then the odds that you'll get a click-fraud bot visiting are increased. If not, then not.

Although it's quite possible that ads not in the Adsense program are getting targeted by these click-fraud bots. But in that case you'd never know since the other networks are probably not as sophisticated as Google is and just coughing up the extra bucks. The publishers profit as an after-thought.