Google analytics data is getting less accurate as the days go by

Google analytics blockers are on the increase

[davyeminy]

Google analytics blockers are actually higher than I thought.
Yesterday, I received two messages through a contact form I placed on one of my pages, only to view my analytics data and realise that these users used browsers which block Google analytics! I get majority of my traffic from the U.S, Canada, Germany. This, is not the first time that this is happening. I also realised that Google analytics can't track users who disable JavaScript in their browsers, because I've done the experiment myself before.
Now, I use trending posts function which displays the most viewed posts for the day. I also realised that some posts that shows as trending don't show up in my Google analytics data. I've tried using cloudflare to estimate by blocking all bots from getting to my server only to realise that analytics data was far inaccurate. Yesterday, I received half of the traffic I received the previous day according to Cloud flare but my analytics says that the traffic received was far far below half at about a 70% decrease!
I'm outrightly confused because I'm loosing trust in the data I'm getting in my analytics data.
Someone please help me out, what could be the problem? Are people blocking analytics Data this much?
I'm using monster insights WordPress plugin.. Could there be an issue with this plugin?
Is there a way I can manipulate my tracking code to prevent it from been blocked
Thank you.

[lucy24]

And now the bad news: Analytics of any kind--whether third-party like GA, or even something like piwik/matomo that lives on your own server and is subject to your own access controls--can never and should never be used as your sole information source for what is happening on your site. Sure, look at analytics. But also look at your server access logs. Combining two types of information from two sources will start giving a more accurate picture.

I also realised that Google analytics can't track users who disable JavaScript in their browsers, because I've done the experiment myself before.

Does GA not have a <noscript> track? Sheesh. (And why did you need to experiment? I'm all for hands-on learning--it's why I maintain a test site--but this is like stepping outside in the rain to verify that you will get wet.)

[Dimitri]

Someone please help me out, what could be the problem? Are people blocking analytics Data this much?

Yes. With the increase of concerns and issues about privacy over the Internet, tracking, and so on, this is the trend since some years. North America is not "immune" to this trend. All browsers have different kind of privacy protection features, and you have all the add-ons which are popular, and blocking third parties codes, cookies, etc...

So people (browsers) blocking GA is not a "problem", it's an answer to the problem of privacy protection of Internet users.

Is there a way I can manipulate my tracking code to prevent it from been blocked

It's never a good idea to go against the will of Internet users visiting your site...

The solution would be to set up your own stats system.

[yaashul]

Hi,

I do not know such a sharing is allowed or not. There is a discussion on cloudflare and a user boynet has created a cloudflare worker script to tackle this issue - [github.com...]

[Edge]

In general ~ folks who java block are unlikely to ever convert into a customer. And... if you don't care that they don't become a "customer" then server stats showing unique pageviews less bots would be the yard stick.

[NickMNS]

I've tried using cloudflare to estimate by blocking all bots from getting to my server only to realise that analytics data was far inaccurate.

Cloudflare reports "Requests", "Bandwidth" and "Unique Visitors". Requests include every single request made to the server for every single asset, every time a page is loaded. Bandwidth is is bandwidth is consumed by all these requests and "Unique Visitors" include every single visit to your site by both bots and humans a like, without differentiation. There is a lot of bot traffic on the web, most of it is not bad or malicious and is in fact necessary (eg: Googlebot, Bingbot, mediaPartners (Adsense) etc...). Blocking all bots indiscriminately is not advised, and will certainly have a negative impact on your websites performance in search.

Most bots are programmed not request/trigger Google Analytics (GA). Most of the traffic reported in GA is human (but not all).
GA reports Sessions, Users and Page Views. The closest metric to Cloudflare is "Users" which coincides with "Unique Visitors" but as mentioned above GA doesn't include bots. Page Views is somewhat similar to requests, but a single pages view may include dozens of individual requests, so the values reported will be off by an order magnitude.

Bottom line is Cloudflare vs GA is apples and oranges, they both provide valuable insight in their own right but it is difficult to compare the data.

[NickMNS]

@Dimitri

It's never a good idea to go against the will of Internet users visiting your site...

Tracking the presence of users on my server is not going against the will of the internet user. It is like saying that one shouldn't be allowed to refuse entry to people that show up to your brick and mortar store wearing a ski mask saying they do not want you to track them.

[Dimitri]

I just meant that, if Internet users installed / set up something to block tracking, then its their will not be tracked... and here it's about GA, so it's not only a matter of tracking users on your site, it's tracking user all across the Internet ...

[Achernar]

I'm using chrome with uBlock, and I can tell you that it filters google analytics by default. It even has techniques to prevent sites preventing it from doing it. It has been doing this for a very long time, so I don't know if it is the source of your problem.
What I've read here on another subject about firefox, is that the lastest version has tracking protection enabled by default. [webmasterworld.com ]

As lucy24 has written, you're better using your server's access logs.

[Dimitri]

you're better using your server's access logs.

+1
I am old school, so I never understood why a web"master" needed a third party to tell him(her) who is visiting what (just teasing, no hard feeling :)

[JS_Harris]

For $5 on a couple of popular jobs sites I can buy 10,000 "visits" that analytics will not be able to detect are bots. It comes in spread over time, over countless IPs, countries, languages, some from search, social, 3rd party sites in very believable proportions, different monitor sizes.... this thing was built to fool analytics and it does.

For an extra $1 I can get 10k more as an addon job.

I wrote about this a while back when I realized you can NOT go by analytics reports when buying a website anymore. I know some of you more seasoned webmasters will be thinking "I could tell", trust me, you most likely could not as it's very human-like in EVERY metric incl bounce rate, pages visited(yes it visits pages from other pages). Probably the ONLY way you will detect it is by noticing it won't leave via adsense. It will, however, leave through other external links.

I'm not plugging these job sites, in fact many make the same promise on those sites, but this one particular setup is on another level.

[JS_Harris]

Additionally, re "blockers".

I run some myself including the popular ublock origin. I mention that because the block rates are not what you'd expect. It comes with default lists and those lists are actually very friendly to Google's analytics and adsense. One of them has them both whitelisted.

In contrast three default lists bury the adsense competitor media.net much more harshly. The blockers are tougher on Google's competition by default than on Google. I don't know if that's earned or paid but I can confirm it's true right now.

[Shaddows]

GA is blocked in hosts (with a loopback), and via noscript on my machine.

Almost all my tech savvy friends operate noscript, adblock plus, or a bespoke hosts list, though Google-blocking is not necessarily the reason (especially for ABP).

On any given day, our GA-reported revenue is between -5% to -15% compared to real revenue. This has drifted upwards from 2-6% since 2012. We're B2B for physical product (services sometimes attached) in a tech-savvy niche. (This is actually one of our "update-is-coming" metrics, where a change in distribution presages the update)

[Achernar]

JS_Harris: I've verified with a clean uBO config, and GA is filtered by PL’s Ad and tracking list. But maybe it is whitelisted by another list for a couple of sites.

[graeme_p]

@Dimitri, there are a number of reasons people use Google Analytics:

1. Customers like it
2. If you ever want to sell a site its what potential buyers ask for
3. It does a lot more than any access log based tool I know of.
4. You can set it up to keep GDPR compliant stats by not recording the full IP.

If I am wrong and there is an access log based alternative that even comes close (on 3 and 4 at least) please tell me because I will use it on my own sites in an instant.

[NickMNS]

Let me add one important point to Graeme_p's great list.

It's free. Not just in the sense that you don't need to pay for it up front, as there are many other solutions that are free and/or open source. It is free in the sense that it doesn't run on your server it runs on an external resource and does not consume computing resources such as memory and cpu. There may well be log-based solution available but one would need to provide the resources to run it. Providing an equivalent solution to GA would be no small task.

[lucy24]

it doesn't run on your server it runs on an external resource and does not consume computing resources such as memory and cpu

Heh. And for some users, this is one of the leading disadvantages.

Analytics programs don't--or shouldn't--consume any server resources except when you are actively using them. If you only look at analytics once a month, your server is only doing extra work once a month. The rest of the time it's just logging requests, and that's trivial compared to all the other requests that accompany a human visit. In fact, when I process my raw logs, I concurrently pull out requests for the piwik file and subject them to some simple processing. Quicker and simpler than going online loading up the whole package.

[iamlost]

GoAn as well as Matomo/Piwik are relatively simple and easy to use, as well as 'free', which is why they, GoAn especially, are the most common website analytics programs.

However, they are neither as accurate nor as, potentially, data rich as log files. Unfortunately, most software that leverages this power is enterprise level, which means there is a learning curve, and often expensive, which means not practicable for many. And may require programming (C/C++, Java, Python, SQL...) for customisation to meet site specific requirements.
Note: just as Google organic search referrals and AdSense are simple easy defaults and moving beyond them has a learning curve.

If you do want to look into moving beyond javascript analytics a [ open source log file analysis ] search will get you started on your research.

An actual ETL (Extract, Transform, and Load) analytics process is yet another step up. And yes, GoAn can be part of such a methodology if one wants to ease on up. The current best off the shelf open source version is probably the 'ELK Stack' aka Elasticsearch + Logstash + Kibana. I'm 'lost', i.e. been playing with this stuff for quite a while so my system, several years old, is cobbled together from Hypertable, Quantcast File System, MapReduce-MPI Library, Postgres-XL, Redis (especially streams)... What one can do with analytics at these levels is implement their findings in real time as well as utilise machine learning stored data to check real time heuristics.

GoAn and it's ilk are but the entry to a huge analytics capability space. It's really a matter of determining your site analytics requirements and seeking the best available tool to meet those. Sadly, too many choose the tool and then are constrained by it; cart before horse.

[tangor]

@davyeminy Welcome to Webmasterworld!

Do your own server logs. That's the ACTUAL TRAFFIC ON YOUR SITE.

If you can't code your own analysis, use the many free log analysis programs out there to help refine the data.

My custom code (databased and query driven) is pretty severe. And in my experience truly identifies HUMANS far better than other products. This code has been fine tuned over a 20 year period. As a result, reporting on a site that gets 100k hits a week reveals 2.8k came from humans, and reveals how many pages each UNIQUE visitor, and how long on site.

No js on site (ie, no g analytics). Personally I surf with JS off ... AND with script blockers/ad blockers active.

But one must bear in mind that "logging" and "analysis" is constantly evolving. The web is getting noisier with automation (bots, scrapers, hotlinks, etc) and many of the current tools just haven't kept up.

MEANWHILE, use your server log to benchmark the reports you get from other parties (such as Cloudflare, g) to further interpret the "reality" of traffic on site.

One can never have too much data!

[Dimitri]

And don't forget that, one day, Google can decide to charge a fee based on your volume of traffic, like for Google maps :) ... If, one day, Google can no longer monetize the data they are collecting through GA, they will need to find another way to cover their expenses ...

[fearlessrick]

G Anal never worked for me at all. I found it more of an annoyance than anything else. Data was not just skewed, it was indecipherable.

Thanks to everyone for my daily geek-fest. Way over my head.

[graeme_p]

@imamlost is probably right that ELK is the best alternative.

Its a bit of a pain to install.

I looked at it for error logging and ended up going with Graylog for that, but ELK seems better (out of the box anyway) for analytics. Has anyone used it?

I am somewhat templated to set up an install for my customers but I think access controls are a paid feature (still?) so they cannot share an install and there is still the what the customers are used to/prefer problem.

[JS_Harris]

Alternatives? Sorry, stop.

Analytics has crapped out, it's unable to tell human from bot, period. You don't need an alternative to remove it, just take it down just like you would any worn out part on your car. The site will run just fine(probably faster) without it. Your visitors will enjoy a bit more privacy at the same time and you'll be starving the aggregators that monitor(spy on) your analytics UA in the wild. You'll also have more free time to create on your hands, many spend hours looking at analytics per week.

Raw server logs and a competition metrics site like Moz, SEMrush, SERanking (etc) will provide you with months of content creation ideas. Google's search console reports on your site's results(for free) if you need to see but neither it nor analytics ever tell you what your competitors are up to.

It's not yourself you need to look at anymore, it's your competition.

[cnvi]

Agreed.. I used to use WebTrends years ago.. can anyone suggest a user friendly server log reporting tool?

[csdude55]

Is Urchin still a thing? I used to use that several years ago, but moved to Analytics because it was taking up so much server storage. But like others, I've found that many ad blockers are blocking Analytics, too. Since I use it to sell ads locally (photo text ads that wouldn't be blocked by the ad blocker), that's an issue for me :-(

It's helpful for me to use as a guideline to know which pages / features are most / least popular, but that's about all. I can't really rely on it for anything else.

[azlinda]

I can attest that it is unreliable. GA shows my CTR for the past three months at an average of 3.4%. In reality, I have not seen anything over 1.0% for the past three months.

[blend27]

Ya'ol mean Truth is NOT truth? ;)

Urchin got disinfected way back when G put its fingers into it.

I run a live server/client side script that was written more than a decade ago. SS logs are away to go if you looking for that real truth.