Welcome to WebmasterWorld Guest from 54.198.106.21

Forum Moderators: incrediBILL & martinibuster

Adsense support says earnings reduced due to invalid traffic!

     
11:35 pm on Jul 11, 2017 (gmt 0)

Preferred Member

10+ Year Member Top Contributors Of The Month

joined:Dec 19, 2004
posts: 554
votes: 3


Hi so my site's Adsense earnings has reduced by about 45% in the last few weeks!

So, I opened a trouble ticket with Adsense and they are saying that my earnings are being deducted due to traffic coming from "bad sources"!

I've placed my site on Cloudflare ( a reverse caching proxy) and I am whitelisting only these IPs as valid origin IPs as per Cloudflare's list:

103.21.244.0/22
103.22.200.0/22
103.31.4.0/22
104.16.0.0/12
108.162.192.0/18
131.0.72.0/22
141.101.64.0/18
162.158.0.0/15
172.64.0.0/13
173.245.48.0/20
188.114.96.0/20
190.93.240.0/20
197.234.240.0/22
198.41.128.0/17

So my question is how to fix this issue?! I am in dire straits so any help/suggestion would be greatly appreciated. What steps can I take to find source of bad traffic so as to ban them and help increase my earnings to normal?

Thanks!
11:52 pm on July 11, 2017 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:8995
votes: 409


So my question is how to fix this issue?
My opinion? Move your site off Cloudflare, at least for a couple months to test and see if that resolves your issue.

Similar issues concerning Cloudflare and other CDNs come up all the time.
12:12 am on July 12, 2017 (gmt 0)

Full Member from NZ 

10+ Year Member

joined:July 28, 2003
posts:308
votes: 6


Any clicks are on the client browser and those report their actual IP addresses. Being behind Cloudflare has nothing to do with it as this is just the path the client browser uses to request the pages and elements within. Cloudflare or no Cloudflare, clicks still come from the client browser.
1:15 am on July 12, 2017 (gmt 0)

Senior Member

WebmasterWorld Senior Member Top Contributors Of The Month

joined:Mar 17, 2015
posts:652
votes: 343


I would run some tests if I was you. In theory a CDN should not affect Adsense traffic or earnings, since it's the visitor's IP that determines geotargeting, validity, etc. But a lot of people do report drops in CTR and revenue after shifting to CDNs. Some of that may be down to poor configuration, e.g. using tools and plugins that mess with JS. But it's all anecdotal so who knows for sure.
1:19 am on July 12, 2017 (gmt 0)

Preferred Member

10+ Year Member Top Contributors Of The Month

joined:Dec 19, 2004
posts: 554
votes: 3


So it's invalid traffic too could there be bots or some spam originating IPs from nations? Please help thanks!
1:26 am on July 12, 2017 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:8995
votes: 409


Yes, bots are a likely candidate for invalid traffic with Adsense.

But if you use Cloudfare can you get accurate & complete daily/hourly raw server logs to determine which agents are responsible?
1:55 am on July 12, 2017 (gmt 0)

Preferred Member

10+ Year Member Top Contributors Of The Month

joined:Dec 19, 2004
posts: 554
votes: 3


Yep there is mod cloudflare an apache module that'll log the real ips into access logs.

Then how do I identify which ip is bot or spam origin?
2:03 am on July 12, 2017 (gmt 0)

Senior Member

WebmasterWorld Senior Member Top Contributors Of The Month

joined:Mar 17, 2015
posts:652
votes: 343


There's a few things you can do. The first thing is to identify traffic spikes from countries where bots are likely to originate, e.g. Russia, China, Ukraine, Romania. Also increases in suspicious referrals. There are numerous threads on bots on WW.
2:16 am on July 12, 2017 (gmt 0)

Preferred Member

10+ Year Member Top Contributors Of The Month

joined:Dec 19, 2004
posts: 554
votes: 3


So I've added captcha challenge for all ips from Russia and Ukraine via cloudflare so let's see. Should i add China too? Any service that analyses accesslogs to identify bots etc for a fee? Thanks!
2:29 am on July 12, 2017 (gmt 0)

Full Member from NZ 

10+ Year Member

joined:July 28, 2003
posts:308
votes: 6


That's a good move. You could even (if you have coding experience) not show ads for visitors coming from these IP addresses.
3:05 am on July 12, 2017 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:8995
votes: 409


Why challenge these regions? Stats show most bots come from the US and next is Europe.
4:12 am on July 12, 2017 (gmt 0)

Preferred Member

10+ Year Member Top Contributors Of The Month

joined:Dec 19, 2004
posts: 554
votes: 3


You are absolutely right keyplyr.. i was checking traffic (bot) stats in Cloudflare just a few mins ago.. most were from USA! not good! Are these other countries using vpn etc to hide their bots?
4:22 am on July 12, 2017 (gmt 0)

Preferred Member

10+ Year Member Top Contributors Of The Month

joined:Dec 19, 2004
posts: 554
votes: 3


Infact now (1-2 minutes) I'm detecting an ip from Taiwan trying ddos on my site.. sad! Perhaps this is after I blocked China... an hour ago.. so it's a mess! :-(
4:34 am on July 12, 2017 (gmt 0)

Preferred Member

10+ Year Member Top Contributors Of The Month

joined:Dec 19, 2004
posts: 554
votes: 3


Anyways blocked that ip custom rule.. perhaps my Adsense earnings will show improvement? Thank god for google notification! I opened a ticket with them!
5:24 am on July 12, 2017 (gmt 0)

Preferred Member

10+ Year Member Top Contributors Of The Month

joined:Dec 19, 2004
posts: 554
votes: 3


Yep went to [spamhaus.org...] and figured from there which countries are the most bot originators.. installed captcha challenge on all! I never released this problem existed in my site!

Also, if I don't use cloudflare then how to do country blocks? Any suggestions?
5:46 am on July 12, 2017 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:8995
votes: 409


I never block countries or regions. It's easy to get the impression that one country or another is all bad actors, but that's never the case. Actors can use ranges from anywhere.

You need access to your hourly/daily raw server logs to develop a comprehensive strategy of defense using various Blocking Methods [webmasterworld.com]
6:07 am on July 12, 2017 (gmt 0)

Preferred Member

10+ Year Member Top Contributors Of The Month

joined:Dec 19, 2004
posts: 554
votes: 3


Keyplyr you are right, it's a temporary captcha challenge I've placed. Let me see how this affects my traffic and earnings then I shall consider your advice.

This bad quality traffic seems to have affected referral traffic coming from Google search as well! :-(
6:57 am on July 12, 2017 (gmt 0)

Preferred Member

10+ Year Member Top Contributors Of The Month

joined:Dec 19, 2004
posts: 554
votes: 3


So I googled and found sites where you can add filters to Google Analytics to remove bot traffic.

I have the reverse question.. can Google Analytics let me know which countries are sending bot traffic? That is, does it show any data on illegal bot traffic that can be analysed to see which countries they are originating from? (I know these hackers can fake countries but this is a start I guess.).

Thanks!
7:04 am on July 12, 2017 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:8995
votes: 409


3rd party filters do not work because they block with a broad brush. They do not allow the good traffic through from the ranges they block. Read the articleI linked above.

There's no substitue for learning.
12:30 am on July 13, 2017 (gmt 0)

Preferred Member

10+ Year Member Top Contributors Of The Month

joined:Dec 19, 2004
posts: 554
votes: 3


So now my google search referral traffic has gone down too! Would this happen coz of invalid traffic too?

Whole mess! Please help.
5:01 am on July 13, 2017 (gmt 0)

Preferred Member

10+ Year Member Top Contributors Of The Month

joined:Dec 19, 2004
posts: 554
votes: 3


So also, there are countries that have page impressions (adsense reports) in the say, 21000 page views in a month, and zero clicks.. should I filter/captcha these countries too?
5:22 am on July 13, 2017 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:8995
votes: 409


As I said earlier, filters will block valid users
and CAPTCHAs will cause many users to just go elsewhere. People hate those things. I hear it all the time.

There's no fast fix. Learn how to use correct blocking methods.
7:11 am on July 13, 2017 (gmt 0)

Preferred Member

10+ Year Member Top Contributors Of The Month

joined:Dec 19, 2004
posts:554
votes: 3


Agreed also can i use g analytics to find out useless traffic?
7:45 am on July 13, 2017 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:8995
votes: 409


Sure, but GA misses quite a bit. It has its uses, but falls short when it comes to correctly identifying bots and actual humans.

Having all the GA code on every page also slows page load time and often hangs up the page for users on slow connections. If you already use a couple Adsense units, you may not want to add yet more 3rd party http connections.

Analog [mirror.reverse.net] is free and a robust traffic analyzer. No code is needed on your pages. It runs on your local machine and reports using your downloaded raw access logs. It's highly customizable, reporting only the metrics you want. I've been using it for years.
8:25 am on July 13, 2017 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Feb 12, 2006
posts:2623
votes: 78


if you're desperate you could try installing something like the free "bad behavior script" (search for it on the web), which blocks a lot of bots before they even see the page. it takes a bit of fiddling with to get it working though, to set up the database, because their docs aren't that great
8:36 am on July 13, 2017 (gmt 0)

Preferred Member

10+ Year Member Top Contributors Of The Month

joined:Dec 19, 2004
posts:554
votes: 3


Ok this is what I decided based on advice here and some logs from Cloudflare.

I removed all country's captcha challenge first of all. Coz I checked with GA that many visitors are from google search from the same countries known for bots and stuff. So I agree with keplyr that this captcha for Google search visitors will be irritating.

Secondly I increased the security level on cloudflare so as I see the traffic logs in Cloudflare it's validly blocking users who use DDOS bots or fake UA bots etc.

I'll be checking the logs again and again to fine tune these settngs. Let's see what effect it has on Adsense and G serps etc.
8:49 am on July 13, 2017 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:8995
votes: 409


To learn more about bots and other user agents (UA) read the posts in the Search Engine Spider & User Agent ID Forum [webmasterworld.com]

To learn more about what IP ranges are candidates for blocking, read the threads about Server Farm IP Ranges [webmasterworld.com]
12:45 am on July 18, 2017 (gmt 0)

Preferred Member

10+ Year Member Top Contributors Of The Month

joined:Dec 19, 2004
posts:554
votes: 3


If i block site visitors wo UA then some legit bots would be blocked too? Won't they?
12:59 am on July 18, 2017 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:8995
votes: 409


This is not a quick process. It takes many years, and new agents come almost every day.

Read through your hourly/daily raw server logs several times a day, each and every day... research every IP and User Agent that shows abnormal behavior.

Read the posts in the Search Engine Spider & User Agent ID Forum [webmasterworld.com] to learn about good & bad bots.

Block Hosting and Server Farm IP Ranges [webmasterworld.com]

Continue to read through your logs and research everyone blocked to see if you are blocking any good bots or humans, and if so, allow exceptions.
1:56 am on July 18, 2017 (gmt 0)

Preferred Member

10+ Year Member Top Contributors Of The Month

joined:Dec 19, 2004
posts:554
votes: 3


Edit: I posted in the apache server.

Also no use as cloudflare changes all 1.0 requests to http 1.1 requests
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members