Coverage dropped below 70%

[ruggiero]

Just as the title says coverage dropped below 70% by the end of December and it still hasn't gone up. I have 2 responsive ads and matched content with ads. I see blank spaces frequently and strangely matched content doesn't show sometimes either. I'm not talking about the ads inside of it but the entire thing. Isn't matched content supposed to show all the time?

Please help because my revenue has dropped by 50%-75%!

[AlexB77]

@NickMNS, right you are.

Have you checked your coverage on country by country basis?

[NickMNS]

I just checked now, US has been a little higher than other countries but all are in the same order of magnitude.

[dk385]

Did anyone recovered from ad coverage drop?

[azlinda]

Did anyone recovered from ad coverage drop?

Not I. It's still bad.

[dk385]

Same with my sites

[keyplyr]

Did anyone recovered from ad coverage drop?

Yes... read through this discussion.

[HowYesNo]

doubled traffic, earnings at 50%

so now i'm earning 4x less before ad coverage drop

[danone]

I have an idea I am going to create a new sitemap for static pages where there is million and wait for media-partner google to crawl. My idea is instead of waiting for mediapartners bot to jump link to another. I will show the easy way to find out that. I will let you know result.

[edited by: danone at 1:37 am (utc) on Jan 22, 2018]

[NickMNS]

@danone that is not how the mediapartners bot works. The bot goes to a page every time a user requests it and the page does not already exist in its index.

You can check your logs, you will never see the mediapartners bot ever crawling from page to page all on its own. At least I have never seen it.

[danone]

@NickMNS oh that's sucks then :( do you think mediapartner bot follows googlerobot? I am about to remove nofollow index tags. in this case this is nothing to do with googlebot however mail I receive advice to fetch with googlebot. That's either every employee at adsense have their own ideas they don't actually exactly know what this update about or they are trying to change direction for me to not find whatever the reason is.

[NickMNS]

Adsense documentation says that Mediapartners-bot uses the same cache as Googlebot. So if Googlebot is not caching pages it might take medipartners longer to fetch pages. Also a noindex tag tells the bot not to add the page to the index, now there is much speculation here as to whether or not Mediapartners-bot obeys this or not. I have no idea, there is no indication either way in the docs that I have seen. I would not take the chance, but there are valid arguments for both sides.

[dk385]

@keyplyr

I went through discussion and can't find anyone reported recovery from coverage drop. What am I missing?

[keyplyr]

@dk385,

Several members, including myself, posted that if you browse through the pages on your site with the missing ads, especially using the Chrome browser, then the Mediapartners-Google bot will follow you and reindex each page to Adsense. Doing this has shown to bring the missing ads back to display again on those pages.

This is from experience and assumes this is the scenario. I have not read an official announcement from Google regarding this.

[dk385]

@keyplyr
Thank you for quick response.

I've began, experimentaly crawling my sites with phantomJS and seen improvements but stopped because of possible impression freud.

I have one site with 400,000 pages and one 45000.

Should I continue doing it?

What do you think?

P.S. I have 11 years old adsense account with no invalid click activity ever reported. ( I clicked my ads by mistake 10 times in 11 years)

[robzilla]

That does sound like an activity that could be seen as fraudulent, and is probably easy to detect. Crawling 400k pages like that would not be insignificant, so I would recommend against it; you might end up making things worse for yourself.

[dk385]

@robzilla, Thanks. So i did right thing by stopping.

What options do I have left, that are legit?

[robzilla]

There is simply no way that Google would do this if they had a enough coverage.

You're misinterpreting the YouTube story, which is not about coverage but about the quality of the content that advertisers pay to sponsor. Sometimes you need to make sacrifices, such as losing out on revenue, in order to protect the broader ecosystem and, in this case, prevent advertisers from jumping ship. You lose some money in the short term, but in doing so you protect yourself for the longer term. Unfortunately, some people have a hard time understanding that a company would willingly lose out on revenue this way.

Last I read about the AdSense coverage issue is that they're working on "further guidance in the short term" [marketingland.com] for impacted publishers, and that crawler access is the likely culprit. I'm sure they're reading along here, and hope they'll issue some sort of statement soon. I do wonder it might be related to the YouTube story in the sense that additional quality control is in effect, and that if you're affected perhaps some of your content is difficult for them to label (if it is, indeed, fully accessible to the crawler), or they're now labeling it a poor fit for advertisers. This seems to me much more likely because an inability of the crawler to access your content would probably result in a much larger drop of coverage rates.

[NickMNS]

@robzilla, the link you reference provides no useful information. All it does is paraphrase the email response from AdSense support that was sent to publishers. What is stated doesn't apply. It doesn't apply to me, and doesn't apply to most people affected by this change. Crawler Access is not the issue. And for those that it is, they have hopefully taken the steps to resolve the issue.

If you would like to know more about this issue I suggest starting by reading through the many posts in this thread were the true root cause of this issue has been discussed at length.

@dk385, I'm not sure that crawling your own site is a workable solution. The issue is that the index only has a life of 1 to 2 weeks. This means that you would need to crawl your 400k pages in two weeks in perpetuity. If you have or are crawling pages, be sure to check your logs to see that Mediapartners-bot is actually following you. It is possible that they simply start ignoring request from a specific IP after X requests.

[robzilla]

Depends on what you file under "crawler access". What the crawler does is not just technical retrieval of a document. As the article notes, "the AdSense crawlers categorize, classify and label content for monetization". Changes in the way the crawler is configured to classify content, for example as a result of stricter quality control, are likely to have an effect on coverage, which is why I suggested that possibly your pages are no longer labeled as favorably as before, draining the pool of eligible ads for your website. We're already aware of smart-pricing, where bids are automatically lowered when an ad is expected to perform relatively poorly on your page, but in that case the ad is still displayed, i.e. the brand is connected to the content, just at a lower price. However, if this is indeed about brand protection, as we've seen in the YouTube story, then it makes sense that ad slots stay empty. It's possible they're still in the process of refining the technology, but obviously it's all conjecture at this point.

[NickMNS]

@robzilla respectfully you have know idea of what you are talking about. I will repeat it again. The issue is clear, not a matter of speculation. It was explicitly described in the Adsense Support email that many publishers received. Adwords will not bid on ads that are to appear on a page that has not been previously crawled, or that does not appear in the index at the time of the request. When Adwords pulls out there is no more demand, and the ad slots go unfilled (or it is filled by a 3rd party for pennies on the dollar). That is the issue in a nut shell. For the vast majority of websites, I am sure for yours as well, this change should make little to no noticeable difference. But for a minority of website, such as dk385, mine and others here the impact of this change is devastating. The reason is traffic patterns and life of the index.

From as far as I can tell, from AdSense docs and my data, the life of the index is 1 to 2 weeks, which means that if you have a site that gets its traffic were few visitors that visit many pages and rarely visit the same pages, then those page continuously fall out of the index and each and every request is unfilled. No crawl issues, nothing fancy, plain and simple no if the crawler parameter changed maybe there is change in efficiency that my reduce this or that or whatever.

On December 18 this policy went into effect and overnight my revenue drop more than 60%, while my traffic has been reaching new records. Adsense is doubling down on the "crawl issues", I don't know why. They have now revamped there crawl error warnings. Now I am receiving regular warnings about crawl errors, specifically the inability of the crawler to access the pages I am developing on my localhost. But not a single warning from the pageviews they are actually going to the website. So clearly if crawling were an issue it would have been flagged. But it is not the issue. The issue is a half baked policy that will put me and many others like me out of business.

Robzilla I sincerly respect your opinion as you always bring valid and helpful point to this forum but in this case you are mis-informed.

[kelsheikh]

From as far as I can tell, from AdSense docs and my data, the life of the index is 1 to 2 weeks, which means that if you have a site that gets its traffic were few visitors that visit many pages and rarely visit the same pages, then those page continuously fall out of the index and each and every request is unfilled.

@NickMNS I am like you as well. Lots of URLs that get light monthly traffic compared to others. I have been trying to test what you mention regarding the life of the index but it has been difficult. Specifically trying to find a URL that I know had ads displaying on it 2 weeks ago but then hasn't had any traffic at all till now. Then seeing if ads are displaying. Feel like I'm losing my mind going through analytics.

I'm struggling to find any specific pattern on their index.

[NickMNS]

@kelsheikh the only way to get a clear picture of what is happening is to block 3rd party networks. To be clear, I am not recommending that you do that as this will certainly worsen the situation. 3rd party networks mess up the data because those ads will appear in some situations even when AdWords ads don't or shouldn't.

To find the pages that haven't had traffic over a specific time period is tricky. I was not able to do it with the filters in Analytics I had to export the data and then apply addition filter using Python. You could probably do it in Excel if you are really courages.

[robzilla]

I don't mean to spread any false information, even if just theorizing. Nor am I very much affected by the issue at hand; not in my wallet anyway. Looking at the averages, there's a brief 25% drop on November 10, but overall coverage on my main site has dropped only about 4-5%, with the decline starting at the end of October, and currently at ~94%. I do see some empty slots intermittently, which when filled always display the same ad, whereas the same slot on other pages tend to load a bigger variety of ads. This happens both on popular and impopular pages.

The reason it doesn't particularly affect my bottom line is that the coverage rates only seem to drop for countries where RPMs are low anyway. Most of my revenue comes from Western, English-speaking countries like the US, UK, Canada and Australia, for which the coverage rates have remained at a pretty steady 99+%. Comparing that to India, for example, which sends more traffic than Australia, I see lots of intermittent drops, with November 10 again as the sharpest one, showing a mere 35% coverage rate, which combined with similar patterns for other countries explains the 25% drop I saw in the average rate.

Whatever country they're from, they visit the same pages, so crawler access is not the issue there. Of course, my previous theory regarding changes in labeling of content wouldn't make much sense either. Geography seems to have something to do with it, for me at least.

Have you checked your coverage rates by country? Where's your traffic from mostly?

[edited by: robzilla at 7:41 pm (utc) on Jan 22, 2018]

[glitterball]

As I reported earlier in this thread, my website coverage is great at the moment - but it always is at this time of year (mostly UK audiences).
However, one of my YouTube channels gets a lot of traffic from India, and that channel is currently making 25% of what it was making in November.
Obviously, I don't know about the coverage numbers on YouTube, but that observation does correlate with your theory about geography.

[keyplyr]

There must also be other factors involved with the extreme ad failure some are seeing on their pages.

As I reported earlier in this discussion, I have 2 personal sites each with approximately 300 pages. I only found two pages missing ads beyond the normal occasional unfilled ad space.

Both these sites have pages that are not visited often by users I assume, yet they were not experiencing the extreme ad failure.

[NickMNS]

Both these sites have pages that are not visited often by users I assume, yet they were not experiencing the extreme ad failure.

Can you verify this? It would be interesting to see if anyone is subject to the pattern but not experiencing the impact.

You would need to find a page with AdSense ad code (or pages) that have not been accessed for at least two weeks, then go to the page and see if an AdWords ad is served.

[keyplyr]

Yes, as stated. I no longer have the logs, but the two pages I found missing ads are likely visited every day. Pages that may not get traffic had no issue. This of course is exempting bot traffic.

[SEOPTI]

What's with the adsense quality department guy who responded here, or was it a Google robot?

[NickMNS]

@SEOPTI Who are you referring to?

[ember]

I think he means pubpolicycomms. He said he'd reach out to the appropriate people at Google and get back to us. That was two weeks ago. So either no one got back to him or he's been told not to discuss whatever is going on with the crawling/not crawling/ad coverage, etc.