Adsense support says earnings reduced due to invalid traffic!

[born2run]

Hi so my site's Adsense earnings has reduced by about 45% in the last few weeks!

So, I opened a trouble ticket with Adsense and they are saying that my earnings are being deducted due to traffic coming from "bad sources"!

I've placed my site on Cloudflare ( a reverse caching proxy) and I am whitelisting only these IPs as valid origin IPs as per Cloudflare's list:

103.21.244.0/22
103.22.200.0/22
103.31.4.0/22
104.16.0.0/12
108.162.192.0/18
131.0.72.0/22
141.101.64.0/18
162.158.0.0/15
172.64.0.0/13
173.245.48.0/20
188.114.96.0/20
190.93.240.0/20
197.234.240.0/22
198.41.128.0/17

So my question is how to fix this issue?! I am in dire straits so any help/suggestion would be greatly appreciated. What steps can I take to find source of bad traffic so as to ban them and help increase my earnings to normal?

Thanks!

[keyplyr]

So my question is how to fix this issue?

My opinion? Move your site off Cloudflare, at least for a couple months to test and see if that resolves your issue.

Similar issues concerning Cloudflare and other CDNs come up all the time.

[freitasm]

Any clicks are on the client browser and those report their actual IP addresses. Being behind Cloudflare has nothing to do with it as this is just the path the client browser uses to request the pages and elements within. Cloudflare or no Cloudflare, clicks still come from the client browser.

[trebuchet]

I would run some tests if I was you. In theory a CDN should not affect Adsense traffic or earnings, since it's the visitor's IP that determines geotargeting, validity, etc. But a lot of people do report drops in CTR and revenue after shifting to CDNs. Some of that may be down to poor configuration, e.g. using tools and plugins that mess with JS. But it's all anecdotal so who knows for sure.

[born2run]

So it's invalid traffic too could there be bots or some spam originating IPs from nations? Please help thanks!

[keyplyr]

Yes, bots are a likely candidate for invalid traffic with Adsense.

But if you use Cloudfare can you get accurate & complete daily/hourly raw server logs to determine which agents are responsible?

[born2run]

Yep there is mod cloudflare an apache module that'll log the real ips into access logs.

Then how do I identify which ip is bot or spam origin?

[trebuchet]

There's a few things you can do. The first thing is to identify traffic spikes from countries where bots are likely to originate, e.g. Russia, China, Ukraine, Romania. Also increases in suspicious referrals. There are numerous threads on bots on WW.

[born2run]

So I've added captcha challenge for all ips from Russia and Ukraine via cloudflare so let's see. Should i add China too? Any service that analyses accesslogs to identify bots etc for a fee? Thanks!

[freitasm]

That's a good move. You could even (if you have coding experience) not show ads for visitors coming from these IP addresses.

[keyplyr]

Why challenge these regions? Stats show most bots come from the US and next is Europe.

[born2run]

You are absolutely right keyplyr.. i was checking traffic (bot) stats in Cloudflare just a few mins ago.. most were from USA! not good! Are these other countries using vpn etc to hide their bots?

[born2run]

Infact now (1-2 minutes) I'm detecting an ip from Taiwan trying ddos on my site.. sad! Perhaps this is after I blocked China... an hour ago.. so it's a mess! :-(

[born2run]

Anyways blocked that ip custom rule.. perhaps my Adsense earnings will show improvement? Thank god for google notification! I opened a ticket with them!

[born2run]

Yep went to [spamhaus.org...] and figured from there which countries are the most bot originators.. installed captcha challenge on all! I never released this problem existed in my site!

Also, if I don't use cloudflare then how to do country blocks? Any suggestions?

[keyplyr]

I never block countries or regions. It's easy to get the impression that one country or another is all bad actors, but that's never the case. Actors can use ranges from anywhere.

You need access to your hourly/daily raw server logs to develop a comprehensive strategy of defense using various Blocking Methods [webmasterworld.com]

[born2run]

Keyplyr you are right, it's a temporary captcha challenge I've placed. Let me see how this affects my traffic and earnings then I shall consider your advice.

This bad quality traffic seems to have affected referral traffic coming from Google search as well! :-(

[born2run]

So I googled and found sites where you can add filters to Google Analytics to remove bot traffic.

I have the reverse question.. can Google Analytics let me know which countries are sending bot traffic? That is, does it show any data on illegal bot traffic that can be analysed to see which countries they are originating from? (I know these hackers can fake countries but this is a start I guess.).

Thanks!

[keyplyr]

3rd party filters do not work because they block with a broad brush. They do not allow the good traffic through from the ranges they block. Read the articleI linked above.

There's no substitue for learning.

[born2run]

So now my google search referral traffic has gone down too! Would this happen coz of invalid traffic too?

Whole mess! Please help.

[born2run]

So also, there are countries that have page impressions (adsense reports) in the say, 21000 page views in a month, and zero clicks.. should I filter/captcha these countries too?

[keyplyr]

As I said earlier, filters will block valid users
and CAPTCHAs will cause many users to just go elsewhere. People hate those things. I hear it all the time.

There's no fast fix. Learn how to use correct blocking methods.

[born2run]

Agreed also can i use g analytics to find out useless traffic?

[keyplyr]

Sure, but GA misses quite a bit. It has its uses, but falls short when it comes to correctly identifying bots and actual humans.

Having all the GA code on every page also slows page load time and often hangs up the page for users on slow connections. If you already use a couple Adsense units, you may not want to add yet more 3rd party http connections.

Analog [mirror.reverse.net] is free and a robust traffic analyzer. No code is needed on your pages. It runs on your local machine and reports using your downloaded raw access logs. It's highly customizable, reporting only the metrics you want. I've been using it for years.

[londrum]

if you're desperate you could try installing something like the free "bad behavior script" (search for it on the web), which blocks a lot of bots before they even see the page. it takes a bit of fiddling with to get it working though, to set up the database, because their docs aren't that great

[born2run]

Ok this is what I decided based on advice here and some logs from Cloudflare.

I removed all country's captcha challenge first of all. Coz I checked with GA that many visitors are from google search from the same countries known for bots and stuff. So I agree with keplyr that this captcha for Google search visitors will be irritating.

Secondly I increased the security level on cloudflare so as I see the traffic logs in Cloudflare it's validly blocking users who use DDOS bots or fake UA bots etc.

I'll be checking the logs again and again to fine tune these settngs. Let's see what effect it has on Adsense and G serps etc.

[keyplyr]

To learn more about bots and other user agents (UA) read the posts in the Search Engine Spider & User Agent ID Forum [webmasterworld.com]

To learn more about what IP ranges are candidates for blocking, read the threads about Server Farm IP Ranges [webmasterworld.com]

[born2run]

If i block site visitors wo UA then some legit bots would be blocked too? Won't they?

[keyplyr]

This is not a quick process. It takes many years, and new agents come almost every day.

• Read through your hourly/daily raw server logs several times a day, each and every day... research every IP and User Agent that shows abnormal behavior.

• Read the posts in the Search Engine Spider & User Agent ID Forum [webmasterworld.com] to learn about good & bad bots.

• Block Hosting and Server Farm IP Ranges [webmasterworld.com]

• Continue to read through your logs and research everyone blocked to see if you are blocking any good bots or humans, and if so, allow exceptions.

[born2run]

Edit: I posted in the apache server.

Also no use as cloudflare changes all 1.0 requests to http 1.1 requests