AdSense Message: Impressions from URLs that Haven't Been Crawled

[danone]

My earning dropped 90% due to 15th december brand safety update due to less impression. Do you affected by google adsense 15th of december 2017 update like me? I sent email adsense team and got response

"We detected that your AdX /Adsense account is sending a significant number of ad requests from URLs that are not crawled."

Do you have similar experience? if yes do you have several no follow tag of problematic pages for google robot ?
can you please answer my question ? have you been affected by this update ?

* I have been affected problem still remaining
* I have been affected problem still remaining. I have no follow tag for google search robot (do not confuse with google adsense mediapartners robot).
* I have been affected, I fixed the issue.
* I have not been affected by 15th dec update.

if you already have problem with this update is there anything you done so far ? Can you please explain your solution ?

thank you.

[edited by: martinibuster at 2:53 pm (utc) on Jan 14, 2018]

[martinibuster]

This has nothing to do with Google's recent core algorithm change.

This appears to have something to do with the nofollow tags. That appears to be stopping adsense's crawlers.

[Rasputin]

Danone, the problem seems to be for sites that have lots of pages that are not visited very often. These might be nofollow but on our sites the ads are not shown on pages that are only visited very occasionally, nothing to do with nofollow.
If a page is visited by a 'real person' that seems to tell the adsense robot to visit, so the same page 30 minutes later will then show adverts - too late for the visitor of course, and perhaps they will disappear from that paga agin before the next 'real visitor'.
The big question is...how long the adsense system will remember the page after a visit. If many pages of a site are only visited once a month and adsense forget them after three days, the problem will remain.
Sites with a large number of dynamically generated pages are likely to be most affected.

[NickMNS]

@danone first welcome to Webmaster World.

Martinibuster is right in that this is not a Search Algorithm update, but I doubt that is even what you meant. This email refers to an update of the AdWords bidding algorithm. Basically AdWords will not bind for impression if has not previously seen the page. It works as Rasputin described. The impacts and reactions are discussed at length in these two threads:
[webmasterworld.com...]
[webmasterworld.com...]

[danone]

Thank you guys sorry my email and received email by adsense didn't appear on this forum I am posting my email and response here they suggest me to use google fetch tool which is related how google robot seen your website not medipartners robot seen your website. Yes some page is visited by visitor sometime but who cares ? yes @rasputin that's true other users can see ad if I visited the page 1 day before or 6 hours before. What is behind this update still couldn't understand. Once user visited page it should have advertise no matter what, your question is important but I believe if you can keep your website without ad that is also a big question, I tell you, information on my websites for the people research not crappy content and I don't know what's behind this logic and I am surprise not many people affected on this forum. I have more than 5 million pages. If I remove google no follow tag, there might be problem with google index quality team with more than 5 million pages, possibility unindex all pages including all sub pages from their index, this is the reason I am unwilling to remove google no follow tags, apparently there is no solution now other than shutdown website, It is now generates no more than $4-10 instead of $50-100 every day, It doesn't even cover hosting fee.

Hi Adsense team, Since 18th of december my website impression went down 70%. I already allowed for media partners google robot 
in robots.txt since last 3 years, I don't think there is crawl issue. First I thought this drop due to coming Christmas 
but still continue now.I checked this page http://www.xxxx.com/glencoe-biology-bacteria-and-viruses-nhvweb-Molecular-Genetics-Section-1-DNA.html 
there is 3 ad spot, my friend from new york she sent me screenshot of this page, all 3 ad spot is empty white space and 
other pages too. I tested from sydney where I live, I see 1 or 2 of 3 ad space is blank, it maybe due to my computer 
visiting website cookies. I googled issue and there are many people is also complain about whatever happened on 18th of December 2017 and onwards to. 
Is there any estimate time adsense is going to fix this issue?. If this is adsense crawl robot issue, Is there any tool 
where I can push manually for adsense robot to crawl my website again? I didn't receive any email from adsense. Thank you. 

adsense response

Hi xxxxx,

Thanks for reaching out to our team. My name is xxxx. I'm happy to help you. :)
As part of Google’s efforts to increase brand safety for advertisers, AdWords and DoubleClick Bid Manager have adopted more restrictive bidding on ad requests coming from URLs that are uncrawled. This is necessary to avoid the risk of ads running on sensitive content.

We detected that your AdX /Adsense account is sending a significant number of ad requests from URLs that are not crawled. Below are some of the possible reasons why a URL might not be crawled:

You may be using complex parameters or encoded strings in your URLs that are unique for each visit, instead of sending us the canonical URL, which is easier to crawl
Your URL may represent newly available content which had not been crawled before you sent ad requests. This is transient as your URL will be crawled shortly after your first ad request
You may be sending an incorrect URL to us because you are manually sending an incorrectly formatted URL in your ad request
You may be sending the URL of an iframe with an ad instead of the URL of the content page that hosts the iframe. (This typically applies to larger publishers).
You may have limits on how often they can be crawled (trawler rejects our crawl requests)
To avoid a potential revenue impact from this change, please consider the following best practices for ensuring URLs can be properly crawled:

AdSense Publishers, see:

About the AdSense crawler
How to fix AdSense crawler errors
Display ads on login-protected pages
Give access to our crawler in your robots.txt file
DFP and AdX Publishers, see Crawler Access.

Here are two additional tools that can help identify what adjustments you need to make. 
The Fetch is a Google tool that enables you to test how Google crawls or renders a URL on your site. You can use Fetch as Google to see whether Googlebot can access a page on your site, how it renders the page, and whether any page resources (such as images or scripts) are blocked to Googlebot. This tool simulates a crawl and render execution as done in Google's normal crawling and rendering process, and is useful for debugging crawl issues on your site. 
robots.txt Tester - The robots.txt Tester tool shows you whether your robots.txt file blocks Google web crawlers from specific URLs on your site. For example, you can use this tool to test whether the Googlebot-Image crawler can crawl the URL of an image you wish to block from Google Image Search.
I hope this information helps clarify. After reviewing the information, let me know if you have a follow up question.

Best,
xxxx

[NickMNS]

The policy is clear, no-crawl no-ads. With a no-follow on a page your are guaranteed NOT to have the page crawled.

Your options:
1- Remove no-follow tags and hope for the best in terms of search rankings
2- Use another ad-network that doesn't enforce such a policy. (Media.net is one to consider)
3- hope things will revert back to the way they were (you'll go broke before that happens!)

I am surprise not many people affected on this forum.

Several people have been affected by this in the forum as I mentioned in the post above there are two threads were this is issue has been discussed at length, it may be worth the read.

In general this update (I assume) only effects a relatively small subset of sites. Because it is pattern based. If you have a site where your traffic lands on many pages and each page is viewed by few users your are likely to be heavily impacted. For most sites, most of the traffic lands on few pages and each page has many visits, if you fit this pattern you are unlikely to be impacted by the update. In your specific case since you have no-follow tags you will be impacted regardless of pattern. However, based on your description of your site it sounds like you would be impacted even if there were no "no-follow" tags. (But the impact would likely be less). Also note that url parameters can impact whether the page is considered crawled or not, so sites using URL's with session params, so each user sees a unique url for the same content, can also be heavily impacted.

I am one those heavily impacted by this update and I am working feverishly to find a solution.

[danone]

Hi @NickMNS thanks for your valuable suggestion I've looked links you provide on this forum too. I am gonna start with number 1 which is remove no-follow tag before I try media.net. I am not sure if they are good in CPC. It is strange that new page created visitor from Australia, I can see all ads no blank ads at all, same page I looked online browser from the US where my most visitor come from have no impression at all. I am not really sure if it is related not crawl page if it does why I can see all ads in the first place from another country other than US. It is work for Australia and not for the US it is more look to me inventory issue or something wrong with the system, but I don't know either what it is going on, google left us alone without proper explanation and without giving us pin point answer to solve this issue. it is really frustrating. Whole website millions of page make $2.4 yesterday is a big joke. no policy violation nice content many times they checked. I wouldn't be surprise if I see $1 tomorrow. Let me know if you find a solution I will let you know if I find something.

[NickMNS]

I feel your pain. There are many thing that don't make sense right now, and there is no saying whether all the pain we are feeling is caused by this one single issue.

Also, size doesn't mater it is not how many pages you have that counts it is how many pages are seen by users. That's the way it was, now it's more like how many pages were seen by users that have been previously seen by the Adsense bot...

[danone]

my understanding is if you have a website contains 1000 pages if most visitor visits only 100 pages you will not get benefit other 900 even these pages are valuable for readers just because these pages are seen (ONCE) every 3 months or 6months but it is not valuable for mediapartnergoogle it just doesn't make any sense.

[ariel1238a]

---

System: The following 2 messages were spliced on to this thread from: https://www.webmasterworld.com/google_adsense/4883158.htm [webmasterworld.com] by martinibuster - 6:58 pm on Jan 17, 2018 (utc -5)

---

Is this the end of Adsense for small sites?
In the last month many suffer from a sharp drop in profits..

As part of Google’s efforts to increase brand safety for advertisers, AdWords and DoubleClick Bid Manager have adopted more restrictive bidding on ad requests coming from URLs that are uncrawled. This is necessary to avoid the risk of ads running on sensitive content.

We detected that your AdX /Adsense account is sending a significant number of ad requests from URLs that are not crawled. Below are some of the possible reasons why a URL might not be crawled:

I find it on some site,
Is this the end of Adsense?

[keyplyr]

I've seen very high Adsense returns lately on the several sites I watch. In fact, both December 2017 and January 2018 are much higher than previous years. Hardly "the end of Adsense."

[danone]

@ariel1238a I don't think adsense is finished but I guess this is business and we don't know what they think when they did this update. For them justifying their action maybe it's a secret clean up. Obviously their standard response for everyone is crawl issue and their suggestion given you is excuse to cover what they changed and changing direction for you to not understand situation.

[Edge]

I’ve been using adsense ads since 2003 and just looking at my earnings – sometimes it drifts up and then mostly down. I’ve been on a downward trend since 2014. Despite increasing my monthly traffic, quality page count more than 2 million backlinks and so on I really don’t have any control over my effective earnings from adsense. I’ve always been white hat never had a penalty or a manual action ect. since starting in 2000. For years I chased whatever SEO rumor hinted from whatever source that claimed to help and mostly nothing. The adsense business model is not viable nor are most others. At this time I would not suggest to anybody that they should get a business plan that does not require 3rd party advertising.

[danone]

@edge they fixed my problem do you still experience this problem ?

[Edge]

@edge they fixed my problem do you still experience this problem ?

Not sure you read or understood my comment correctly..

[danone]

@edge I though you are having same problem too once you post here apparently not. just giving out some advice thank you that's fine I misunderstood you.