Adsense, Firewalls, and Windows Servers - Minimize Claw Backs

[gatormark]

I am looking for suggestions on minimizing Adsense clawbacks. For the record, 2021 was a terrific Adsense year, but my clawbacks have been running about 11.5%. I see individuals who have little to no clawbacks. During my "low clawback" years, it would run 6-7%.

My websites are self-developed, from the ground up, asp.net c# websites on Window Servers. I lease my servers, so I have complete control of them.

I currently block certain countries like China, Russia, etc. (the usual suspects) in their entirety, and that has reduced my clawbacks which had climbed to 29%.

However, I am looking to decrease clawbacks even further in 2022.

Any suggestions considering my development environment?

[dolcevita]

If you use CF (Cloudflare) then the FW rules that I use will surely help. Since I implemented them, my clawbacks have been around 2% for the last 9 months.

[edited by: dolcevita at 1:19 am (utc) on Jan 2, 2022]

[dolcevita]

I managed with help of rules below to take down clawback to +/- 2%

Try to apply these rules to CF (Cloudflare) firewall:

- Allow Know Bot bypass (must be first rule above all others)

Field: Known Bots
Operator: equals
Expression: (cf.client.bot)

- Challenge TOR users

(ip.geoip.country eq "T1")

- Challenge China, Russia, Brazil, Indonesia. I have apart rules for each country:

(ip.geoip.country in {"CN"})
(ip.geoip.country in {"RU"})
(ip.geoip.country in {"BR"})
(ip.geoip.country in {"ID"})

- Make new rules Apply these rules by expression for challenge another countries:

(ip.geoip.country in {"TR" "KR" "SG" "VN" "CO" "TW" "EG" "HK" "AR" "TH" "LA" "GH" "EC" "NP" "MX" "ZA" "MY" "BO" "MZ" "AE" "NG"})

- Apply than next rules to challenge corporate IP addresses:

(ip.geoip.asnum in {3223 3561 3842 4250 4323 4694 5577 6724 6870 6939 7203 7489 7506 7850 7979 8075 8100 8455 8560 8972 9009 9370 10297 10439 11588 11831 11878 12989 13213 13739 14061 14127 14618 15003 15083 15169 15395 15497 15510 15626 15734 16125 16262 16276 16284 16397 16509 16628 17216 18450 18779 18978 19084 19318 19437 19531 19624 19844 19871 19969 20021 20264 20454 20473 20598 20738 20836 20860 21100 21159 21321 21859 22363 22552 22781 23033 23342 23352 24482 24768 24875 24940 24961 24971 25163 25369 25379 25780 25820 27257 28753 29066 29073 29182 29354 29465 29550 29691 29802 29838 29854 30083 30176 30633 30693 30900 30998 31034 31103 32097 32181 32244 32475 32489 32613 32780 33083 33302 33330 33387 33438 33480 33724 33785 33891 34305 34971 34989 35017 35366 35415 35470 35662 35908 35916 36024 36114 36290 36351 36352 36666 36873 36887 36920 36970 37018 37088 37153 37170 37209 37230 37248 37269 37280 37308 37347 37377 37472 37506 37521 37540 37643 37661 37692 37714 39020 39326 39351 39392 39572 40156 40244 40676 40824 40861 41653 41665 42160 42473 42708 42730 42831 43146 43289 43317 43350 44050 44066 45102 45470 45671 45815 46261 46430 46475 46562 46664 46805 46816 46844 47328 47447 47588 49349 49367 49453 49532 49544 49981 50297 50613 50673 51159 51167 51191 51395 51430 51731 51765 51852 52048 52173 52219 53013 53340 53559 53597 53667 53755 53850 53889 54104 54455 54489 54540 55225 55286 55536 55933 55967 56322 56630 56934 57043 57169 57230 57858 58073 58305 59253 59349 59432 59504 59729 59764 60011 60068 60118 60404 60485 60505 60558 60567 60781 61157 61317 61440 62217 62240 62282 62370 62471 62540 62567 63008 63018 63119 63199 63473 63949 64245 64484 132816 133296 133480 133752 134451 136258 197155 197328 198310 199653 199883 200039 201011 201525 202053 202836 203523 203629 204196 327705 327784 327813 327942 328035 394256 394330 394380 395089 395111 395978 20248 44901 200904 53057 200532 50968 135822 55293 57286 201200 24549 39458 200000 14576 54290 206898 60117 20448 201553 54825 31472 8556 29119 60476 25532 49949 51698 42442 11274 57345 54817 200019 53342 33569 201983 132425 197395 42699 31698 42612 29311 54527 63213 27175 13209 29140 27223 31659 49834 49693 30152 19133 198414 45201 31981 62605 61280 53332 61147 51109 19234 40438 58797 26978 29748 35974 262990 43021 42695 39704 62899 53281 59615 55761 52335 16973 196827 32647 14992 198968 196745 62071 132869 56106 32911 24931 57669 48896 45481 132509 39839 63129 53370 25048 28747 46433 55051 18570 13955 16535 22903 9823 46945 263032 36536 50986 199733 48825 35914 33552 52236 28855 198347 40728 18120 53914 12586 55720 27640 62563 202118 9290 45887 51050 20068 49485 40374 14415 46873 14384 54555 263237 20773 53918 4851 32306 133229 28216 36236 42210 51248 49815 34649 41562 33260 24220 52347 45486 33182 53055 51290 132225 133120 42776 55799 48446 263093 56732 42399 47385 40539 42244 29302 10929 47549 200147 393326 198171 57773 47583 43472 32338 9166 62082 198651 24725 29067 197902 42418 29097 196645 56110 23535 29869 62756 26484 25926 15189 20401 24679 25128 39756 32400 9412 9667 51294 23052 28099 45693 17881 17669 17918 50926 201634 22611 54641 61102 132071 10207 45577 132070 262603 29883 24558 38279 199997 50465 14120 11235 50655 17019 31240 199481 16862 47161 56784 59791 59677 202023 199990 50872 54839 58936 11230 62310 38894 47172 262287 46260 14442 133143 197648 39451 58922 27589 42400 133393 201597 28997 60800 33322 38001 199129 197372 57752 201670 14244 22152 34541 196678 43198 47625 42331 62049 35295 42311 53589 59705 36791 14160 34432 41062 59135 201630 25260 23108 40281 31590 10532 22720 27357 33070 45187 7595 26481 29713 13926 54203 62651 63128 62838 30849 14987 47577 54334 63916 50915 21217 59816 23273 59632 29452 59795 60739 15919 49313 57879 56617 62088 45179 27597 201702 32740 58667 12617 199847 25642 14567 35278 197914 41079 1442 43620 197439 198313 42705 44398 13909 34745 24958 17971 47143 59854 57682 3722 13647 20450 30235 47205})

- Part 2 rules to challenge corporate IP addresses (not enough space in first one, You must make two):

(ip.geoip.asnum in {55229 7349 33251 52465 52270 45152 8477 198153 52925 61412 262978 53225 41427 53101 41369 35467 59554 52674 24611 48812 40715 201449 52321 29331 201709 53221 198432 51241 19969 56799 26277 58113 28333 42120 6718 20692 17439 132717 9925 132779 42622 6188 40819 24997 38107 36408 57363 46177 62026 61107 58325 19604 23470 39287 12876 60033 13830 60094 37963 31624 26496 48287 132839 204641 35913 39572 32592 23881 198047 14986 17920 32275 50608 199213 262170 201862 43541 24381 10200 14708 27229 48093 42465 7598 30475 23969 9371})

Threat score to challenge anyone that have treat great than or equal than 5 and lower than 50:

(cf.threat_score ge 5 and cf.threat_score le 50)

and fInally in new rule block all with threat score higher than 50:

(cf.threat_score gt 50)

You can chose for JS challenge or Captcha. This kind of rules works, it is safe to use and will probably solve your issue. If you need help let me know.

btw
Of course do not forget as addition to rules above to 'Configure Super Bot Fight Mode' option to challenge bad bots and to allow verified bots.

[gatormark]

@dolcevita

I do not use Cloudflare and am not sure how or why I would use it. I’ll have to research it.

[not2easy]

People use various methods to block unwanted traffic.

I currently block certain countries like China, Russia, etc. (the usual suspects) in their entirety, and that has reduced my clawbacks which had climbed to 29%.

It would help to know how you block unwanted traffic in order to offer ideas.

[dolcevita]

@gatonmark

I do not use Cloudflare and am not sure how or why I would use it. I’ll have to research it.

Cloudflare is great content delivery network (CDN) that will speed up your website, help you fight again various attacks on your website including DDOS and filter bad traffic to your website.
If your clawbacks are around 11% or even more then it is reason more to start using CF.
Pro option for $20 per month is not expensive.

@not2easy

People use various methods to block unwanted traffic.

There are different methods but i have shared my method via CF that really work. The advantage of this method is that nothing is blocked if the visitors from certain countries goes through a JS challenge or Captcha.

The only exceptions are guaranteed threats with a threat score above 50. And this is quite big, but I also gave them a chance up to 50 to pass the captcha.

As I said clawbacks is from the implementation of this method, minor, about 2% (maybe and lower because i have still a couple low traffic websites that are not using CF). Before i had the same problems as @gatormark with clawbacks above 20%.

[NickMNS]

If your clawbacks are around 11% or even more then it is reason more to start using CF.

I just want to challenge this logic. By blocking bots (using CF or other) you will reduce your clawback, but you wont get the "11%" back, rather you will prevent the clicks that are later deducted, so there is no gain to be had. Blocking comes at a cost.

That said, you are required to do what you can to prevent invalid traffic, so not doing anything comes at a risk as well and the higher the clawback value and the more frequently they occur the greater that risk.

There is a free option for Cloudflare, I'm not sure if it will be sufficient to implement all that Dolcevita proposes but it may be. As mentioned there are many other benefits to using CF for speed, and security, they also offer the cheapest rates for domain registration.

Also, bots are not the only cause of invalid clicks. Page layout is another aspect that needs to be addressed, and that across all browsers and devices. Look out for "cumulative layout shift" or CLS, this is when elements are added to the page after the initial render and this pushes the elements around, often just as the user is about to click something. If an ad appears where it shouldn't this can cause invalid clicks. This can be elusive as it may only occur under very specific conditions, eg: slow network, on a specific device using a specific browser. Or, an even more elusive scenario, is when ads are responsive, if a "smaller" or "larger" ad is loaded it could cause other elements to not fall into place correctly.

Note, I use Cloudflare but I'm not actively blocking bots as described by Dolcevita, my clawback is typically around 1%. Others have reported implementing bot blocking, using CF or other methods and it was not sufficient. So this there is no one size fits all solution, you need to try many things. And things change, so what works today may not work tomorrow.

[dolcevita]

@NickMNS

I just want to challenge this logic...

I've been with Adsense since 2004 and until last year I never had a problem with high clawbacks. Clawback was never more than 3%.

The problem started a few months after the famous core update from May 2020, and it lied within a couple months to 23%, which was the reason for proactive action from my side.

And at that time I used Cloudflare, but without the active participation of specific firewall rules that will challenge bots and block bad traffic, because to be honest clawbacks come mostly through unverified bots that do nothing good and if your clawback is not higher than 3% or 4% then i will also not do anything.

The FW rules I shared are based on personal experience and are not set all at once. I first experimented with the threat score and the TOR visitors challenge which brought down clawback to around 14%.
Later came the challenge for certain countries which lowered clawbacks to 11%.
What really made a big turnaround is the challenge for corporate IP addresses which has reduced clawback to below 2%.

In other words, if the human by coincidence was behind the corporate IP address, he will pass captcha challenge and get a 1 month visit until the next challenge, so I don't see anything wrong with the logic of causing suspicious traffic through the Captcha challenge.

I am still convinced that anyone who has problems with high clawbacks will reduce it to below 3% by implementing the FW rules explained above and have real traffic through real visitors instead of unverified bots that do nothing good with their visit to any website except malicious activity, no matter what kind: bot click (what is number 1 reason for high clawbacks), or copying context, DDOS attacks, brute force, etc...

[NickMNS]

@dolcevita

You state that you had a clawback of 23% and then by blocking bots you brought that down to 2%. That is great, and hopefully the tips you have provided will help others do the same. I will certainly comeback to this thread if I find myself in that situation at some point in the future.

But my point is with your statement:

If your clawbacks are around 11% or even more then it is reason more to start using CF.

This statement implies that by blocking bots you are able to increase revenue by 9% (or 21% if you use the 23% figure as the starting number) or stated differently you are able recover that revenue. But that can't be the case if the revenue was deducted because the clicks were invalid, and then you stop the deductions by blocking bots to prevent the clicks. The result is no clicks and thus no revenue so there is nothing to deducted but again also no revenue. Blocking will not directly increase revenue, sure there are plenty of reasons to block bots as I mentioned in my previous post and then some, but a direct increase in revenue is not one of them. To be fair, you may not have meant this with your statement, but it is implied and I felt it worth while to shine some light on it, so that other do not misinterpret it.

This plays into the false idea that Google's clawbacks are taking money that rightfully belongs to the publishers. And your statements (I'm not saying you, just the statement as written) can be misinterpreted to suggest that if you block bots then in exchange Google will give you your money back, and that simply isn't the case, and it needs to be said explicitly because others in this forum (not you or any one in particular) will and have claimed the opposite.

[dolcevita]

@NickMNS

I'm sorry, but I really don't know what gave you the idea that I thought clawback would increase revenue. I don't think I mispronounced, but if I did and if that's how you understood it, then I'm sorry it wasn't my intention.

Namely, I think that high clawback for a long time is primarily a potential problem for the Adsense account itself, since with all other possible problems, it primarily symbolizes in most cases bad clicks due to bots and bad traffic, and this realization does not make the account owner calm. Another depressing thing is that, for example, a payment of $ 4,000 is expected and reduced by 23% to $ 3080, which has a negative psychological effect and it is a normal feeling to try to be proactive and do what can be done to improve such a situation.

Since I was in similar situation, so I am no longer and I know that feeling, all I want is to create a sense of security and less headaches for the account holder.

[dolcevita]

@gatormark
Last month I recommended and advised @TravelMan to try to implement the same suggestions and FW rules I listed here and I just read that it helped him and that clawback this month was a marginal 2% for him.

[puckparches]

I've been using CF for a couple of years now. A few months ago, my clawbacks were around 14% and I start blocking and challenging IP addresses and countries with CF Firewall Rules. For the last 3 months, my clawbacks are down to around 2%.

I didn't use @dolcevita rules, I checked the IP logs and found the countries and IP address spamming my sites and I block or challenge them.