Welcome to WebmasterWorld Guest from 107.23.176.162

Forum Moderators: martinibuster

Fetch as Google

     
7:05 pm on Sep 21, 2018 (gmt 0)

Senior Member

WebmasterWorld Senior Member lame_wolf is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 30, 2006
posts: 3306
votes: 15


I'm not sure if this is the correct forum, but as it's Adsense related...

I am suffering with the advert disappearing fiasco that many seem to be having.

I just tested one of my URL's using the "Fetch as Google" tool in Search Console.
I chose the "Fetch and Render" and came back with "Partial"

When you look at the results, you'll see two screen grabs. One is "This is how Googlebot saw the page:" and the other is "This is how a visitor to your website would have seen the page:"

Scroll down and you'll see "Googlebot couldn't get all resources for this page. Here's a list: " (with a question mark next to it)

Every one that is blocked belongs to Google.

Clicking on the question mark brings up a pop-up box with a link. That link produces a pop-up box with a list of errors and what to do.

For "Blocked" it says...

"The resource's host is blocking access to Googlebot by means of a robots.txt file."

But my robots.txt file isn't blocking anything. Here it is...
User-agent: *
Disallow:


How can I unblock something that I'm not blocking?

(The results of the screen grabs - Google saw the site with no adverts, The visitor would have seen the alternative adverts)
7:08 pm on Sept 21, 2018 (gmt 0)

Senior Member

WebmasterWorld Senior Member Top Contributors Of The Month

joined:Apr 1, 2016
posts:2414
votes: 643


Google is blocking Googleobt from crawling. There is no need for Googlebot to crawl AdSense ads and other Google ressources. There is no way for you to unblock these and there is no need for it either.
7:20 pm on Sept 21, 2018 (gmt 0)

Senior Member

WebmasterWorld Senior Member lame_wolf is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 30, 2006
posts: 3306
votes: 15


Thank you Nick. I even deleted my robots.txt file and tried again. Same thing.
It could be clearer in the results, esp when it's still complaining that it's my robots.txt file that's at fault.

I'm trying to work out why adverts don't stay. I contacted Adsense support but they stopped replying to me. My wife contacted them too as she's having the same problem, then they stopped replying to her too. We've both pointed out that it's related to this...
[support.google.com...] and that Google Tech know about it, but cannot fix it automatically.
7:46 pm on Sept 21, 2018 (gmt 0)

Senior Member

WebmasterWorld Senior Member Top Contributors Of The Month

joined:Apr 1, 2016
posts:2414
votes: 643


The link you posted above points to a reconsideration request form. Do you have policy violations?
7:50 pm on Sept 21, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 891


Don't worry about that Fetch an Render result. It has nothing to do with the missing ads.

As a temp fix, use the Chrome browsee to browse through your pages and reload those where no ad at all appears. The pages where one ad display but some don't may just be an in invenory issue.

My experience has been the adds will eventually all come back if the pages get consistent traffic.

However, Ad serving in general has and will continue to be volatile. There are sever issues.
8:33 pm on Sept 21, 2018 (gmt 0)

Senior Member

WebmasterWorld Senior Member lame_wolf is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 30, 2006
posts: 3306
votes: 15


The link you posted above points to a reconsideration request form. Do you have policy violations?


It's the only form I found where you can contact Google - esp about missing adverts.

When I wrote to them, I was having the issue of adverts appearing, then not appearing. Had it all year, but noticed it more when I started converting the site from html4 to html5

The person I spoke to said...

I reviewed [url removed] and noticed that targeted Google ads are currently appearing on your site. Please see attachment.

However, if you're still having trouble viewing the Google ads on your computer, there are a couple steps I'd recommend taking:
----

Said to clear out cache, make sure there's no adblockers, deleting cookies - I already did all that lot. The email went on to say...

If you're still unable to view the Google ads after following steps 1 and 2 above, reply to this email with a detailed description of the problem and I'll be happy to investigate further. Specifically, could you please provide responses to these questions?

- Can you send me a screenshot showing this problem?
- What browser version are you using?
- What operating system version are you running?
- What is the exact URL that is causing you trouble?
- Does this issue occur in a different browser or on another computer?
---------
I sent screenshots, 3 different browsers, the OS, URLs etc.

Then I received another email...

"We reviewed your site, and our automated systems show that your ad code has not been implemented properly. Please remember that is a violation of our program policies to modify the ad code. "

Remember, your ad code must be taken from your AdSense account exactly, this includes;

The same whitespace and punctuation
The entire AdSense ad code provided
The code formatting
The code can’t have extra HTML tags within it
The variables in the code (such as google_ad_height or google_page_url) cannot have been renamed or changed

The only thing I had done (not knowing it was a violation) was to remove the white spaces, so my code looked like this...

<script async src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script>
<!-- name-here -->
<ins class="adsbygoogle"
style="display:block"
data-ad-client="ca-pub-xxxxxxxxx"
data-ad-slot="xxxxxxxxxx"
data-ad-format="auto"
data-full-width-responsive="true"></ins>
<script>(adsbygoogle = window.adsbygoogle || []).push({});</script>

I have Aspergers and I find it a lot easier if the code is like that. I apologized, spent all day redoing them. I informed them that the whole page is like it, not just their code.
It still upsets me that I've broken a rule, even though many will see it as a minor thing, I don't.

They never replied after that.

I share the site with my wife. She had her own site until she had to close it down due to ill health. So, she has some pages, I have others. Some we share. As it was affecting her too, she wrote and told them pretty much the same thing. We were both polite, never rude, and willing to give any info they may need.

Prior to any of us contacting them, we weren't aware that a violation being caused. It never stopped the adverts appearing before December last year. So, yes we did have a violation, but that wasn't the reason for contacting them via the form.

My wife had better luck than I... that was until she mentioned me, then they stopped replying. It's making me ill as I feel we've both been blackballed by them.

As a temp fix, use the Chrome browsee to browse through your pages and reload those where no ad at all appears.


I've tried that with mixed results. Plus, they don't stay for long, or don't appear on other browsers even hours later when browsed by Chrome.

I've done it on an empty cache as well as a cache that is a day or two's worth of usage. Same thing. F5 or Shift F5 can make advert appear or disappear - depending on the page at the time.

On a side note, I am not receiving WebmasterWorld email notifications on PMs or thread replies. The box is ticked to accept them.
8:35 pm on Sept 21, 2018 (gmt 0)

Senior Member

WebmasterWorld Senior Member lame_wolf is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 30, 2006
posts: 3306
votes: 15


Google is blocking Googleobt from crawling.


It seems to see link units differently. I just tried another URL. The visitor screen showed two alternative adverts instead of Adsense, but showed the link unit advert.
8:58 pm on Sept 21, 2018 (gmt 0)

Full Member

Top Contributors Of The Month

joined:Aug 14, 2014
posts:289
votes: 89


"It never stopped the adverts appearing before December last year."

Wasn't there some BIG change to how Adsense displays on pages beginning that very month? Like pages with little/no traffic will see blank ads until the ad crawler gets called back to them?

I remember there being a thread on here about that back then.
9:07 pm on Sept 21, 2018 (gmt 0)

Senior Member

WebmasterWorld Senior Member Top Contributors Of The Month

joined:Apr 1, 2016
posts:2414
votes: 643


Did this issue start on December 18th of last year? If so you are impacted by the brand safety update.
See discussion here:
[webmasterworld.com...]

See my post at the bottom of page 8 on how to deal with the issue.
9:25 pm on Sept 21, 2018 (gmt 0)

Senior Member

WebmasterWorld Senior Member lame_wolf is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 30, 2006
posts: 3306
votes: 15


Yes guys, it was related to that. I didn't notice till Jan, but wasn't too concerned as I try not to visit the site if I can, and thought it was because of changes I had made. I'd go back later and they would be there.

It wasn't until I changed to https - as well as rewriting the site from html4 to html5 that I was visiting far more often. I wanted to rule out any html validations that may be causing it, plus it would help mobile users access the text files side of things. There were some validation errors in html4 - but that was mainly because of responsive adverts or similar. They never stopped the page being rendered/accessed etc.

I started html5 on the largest section of the site (approx 1400 pages) which is in constant use by the public. The first batch of html5 pages (which all passed validation) were uploaded provided a good number of clicks. It was the best day of the year, and the wife's account wasn't too far behind. As the days went off, the revenue dropped off - even though more html5 pages were being uploaded. I'd check the first batch and there would either be blank spaces (Because no alternative advert was set up) or showed alternative adverts. Sometimes one out of 3 adverts will show.

I've tried everything you can think of - without breaking any rules, and nothing seems to be able to keep the adverts on the page - cache empty or not.

Chrome seems to be showing more adverts than Firefox.

There's this...
[monetizemore.com...]
which is Chrome's policy update, and I nowhere near break the rules. We both value having our Adsense accounts, so there's nothing we would do to jeopardize it, or to piss off our visitors. I've spent 14+ years on that site. I'm not going to do anything to ruin all the hard work and income.
9:31 pm on Sept 21, 2018 (gmt 0)

Senior Member

WebmasterWorld Senior Member Top Contributors Of The Month

joined:Apr 1, 2016
posts:2414
votes: 643


@Lame_Wolf
I didn't read the article you linked I just scanned real quick but it looks like they are mixing up issues in it. I strongly recommend going through the thread I link to. Also I tried to sticky mail you but your box is full.
10:30 pm on Sept 21, 2018 (gmt 0)

Senior Member

WebmasterWorld Senior Member lame_wolf is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 30, 2006
posts: 3306
votes: 15


@NickMNS

Ah, box full... that could explain why I've not received any emails to my sign up email address. I've ticked the box to email notification of new replies. Someone PMd the other day, but no email then either.

I've cleared out some of the messages, so feel free to PM. :)
I've partially read it, and have bookmarked it till my head is a little less stressed.
10:55 pm on Sept 21, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 891


@Lame_Wolf - I agree with NickMNS in that missing ads seem to be mostly a Brand Safety issue. Ads are pulled while either AdsBot-Google or its mobile doppelganger process your pages and qualify them as safe for their brands.

IMO this process takes way too long. One of my sites had missing ads for 4 months. The other 2 sites got consistent ads back after 5 or 6 weeks.

As I said, a temporary fix is to load your pages with Chrome browsers. It will bring the ads back but they will eventually fall out of the Brand Safety cache, so this is not a permanent solution.
11:07 pm on Sept 21, 2018 (gmt 0)

Senior Member

WebmasterWorld Senior Member lame_wolf is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 30, 2006
posts: 3306
votes: 15


Hi Keyplyr,

That's what I have been doing, but it doesn't seem to always work. I looked at my log files the other day, and picked out a few URLs that were picked up by mediapartners. Not all of the pages showed adverts or a full set of adverts. Not much better than pages that hadn't been checked. (The pages are years old, plain old html, no ajax etc.)

I've not looked at Chrome for the last few hours as I'm trying to rewrite 1000's of pages, but when I did, Chrome was showing more adverts than FF.
11:34 pm on Sept 21, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 891


[Chrome] it doesn't seem to always work.
Agreed and it's task intensive, a real PITA if you have a lot of pages.

As I said, the ads are all back across my sites, but for those difficult pages I often use Chrome a couple times, then use FF or Edge a couple times, then come back and use Chrome. The ads would *usually* start to appear again.

Another tactic I used was to access my pages with a VPN. I suspected that it wan't wise to redundantly load my own pages with the same IP address I was logged into Google, and clearing cookies can be a hassle.

Anyway, from my experience, the good news is that eventually this gets resolved and the ads stay.
11:58 pm on Sept 21, 2018 (gmt 0)

Senior Member

WebmasterWorld Senior Member lame_wolf is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 30, 2006
posts: 3306
votes: 15


It's good to know that it's fixable. I am wondering if it is also linked to my site no longer being cached in Google. It's been like it since I changed to https. If I click on the "cached" button in the SERPs, it says "404. That’s an error. "

If I alter the URL in the cache URL from https to http, it does show a page - minus the css and any images. If that Google cannot cache the page, then maybe Brand Safety cannot be cached either.
12:30 am on Sept 22, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 891


It's doubtful the caching issue is related to Adsense IMO.

Sounds more like an indexing issue regarding the new HTTPS property. There are a few related things to check with that:

• 301 working properly ?
• Is access blocked for any page ?
• Did you add the new HTTPS property in GSC ?
• Did you submit an updated (HTTPS) sitemap.xml ?

Note: Re-indexing HTTPS pages can take a long time for some sites, especially if there are a lot of pages & if you have a lot of pages without backlinks.
2:14 am on Sept 22, 2018 (gmt 0)

Senior Member

WebmasterWorld Senior Member Top Contributors Of The Month

joined:Apr 1, 2016
posts:2414
votes: 643


It's doubtful the caching issue is related to Adsense IMO.

I am not so doubtful. MediaPartners-Google and Googlebot are said to share the same cache. My understanding is that media-partners will look first in the cache if it finds the page, then it
doesn't need to get it for itself, in order for it to add the page to the index. If no pages are cached it could make it much more difficult to get pages indexed. Brand safety == page not indexed ads not shown (AdWords ads 3rd party ads may still be shown).
2:17 am on Sept 22, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 891


MediaPartners-Google and Googlebot are said to share the same cache.
Possibly, but I think it is AdsBot-Google that checks for page quality related to Brand Safety. [support.google.com...]

At one time I also thought it was MediaPartners-Google that validated for Brand Safety because it seemed to follow Chrome around when I checked my logs, but there is evidence it is actually AdsBot-Google. The final validation may be a 2 step process involving both bots.
2:43 am on Sept 22, 2018 (gmt 0)

Senior Member

WebmasterWorld Senior Member Top Contributors Of The Month

joined:Apr 1, 2016
posts:2414
votes: 643


I don't know you could be right about the two step process. But I just checked a site that I am involved with (not mine) that has been having issues since December at the cached version of the pages are returning 404's. I really think that we may have stumble on to something here.
3:17 am on Sept 22, 2018 (gmt 0)

Senior Member

WebmasterWorld Senior Member Top Contributors Of The Month

joined:Apr 1, 2016
posts:2414
votes: 643


I looked into this 404 cache thing a little more.
See my comments in the thread specific to that topic.
[webmasterworld.com...]
Basically I think Keyplyr is right I doubt that it is related simply a bug on Google's end.