Is it a problem, if the Adsense code does not appear in the HTML code

[Dimitri]

Hi

I am adding Adsense (or other ads) to my pages, using Javascript, from an external file (dynamically generated in PHP), which is disallowed in the robots.txt

As a result, there is no trace at all, of the Adsense code, in the HTML code of the page.

Can it be a problem (nowadays) ?

I assumed that, this is the same as when some publishers are using third part services to select which ads to display, between Adsense and other sources.

I never had problem doing like that, but, as I said, I discovered that, within the last year, my Adsense earning collapsed, (divided by 10!), whereas my traffic increased by 15% during the same period. I know that a lot of things changed while I was gone. I suspect the Brand safety protection has something to do with that, so I am trying to investigate everything.

So I was wondering if, this can be an issue. When Googlebot, or the Adsense bot visit my pages, they won't find the adsense code. Also, if they run Javascript, they may not find the code either, since it's inserted from an external file, which is disallowed in the robots.txt. So can it have an impact ?

In the other hand, I see ads being displayed, and still earning money. If there was a problem with that, I guess nothing will show at all.

add: also, I find that the adsense bot's activity on my sites to be very low, compared to what it used to be. One year ago, the Adsense bot was fetching like 5.000 pages / day, whereas now, it's more like 200-300.

[NickMNS]

Can it be a problem (nowadays) ?

Nyes (that is no and yes)
Implementation is everything in this case. Done correctly it should work fine, done incorrectly it could cause you some serious issues including getting you account band.

First off, you must not use (or be very careful of how you use)document.write() or any js that uses document.write() under the hood, like the jQuery command $().html or the plain vanilla innerHTML. For example if you are appending content to a "Div" element and you use document.write() the content of the div will be copied and then re-inserted into the DOM with the new content appended, but the original will remain on the page, hidden beneath the new copy. If you have an ad unit that is part of that content the ad unit will be copied as well and you will have one ad-unit covering another. This is not a good situation. Moreover, it is not easy to detect as it simply appears as if the ad was refreshed (more about refreshing in a bit).

Note that the above situation will inflate your page views and reduce you CTR and likely cause your AVV to fall. If you are adding content multiple times to a page this impact is compounding. Each existing ad unit will be copied for each click or scroll to ad new content, so a single page could end up with dozens of ads on it of which only the top layer is visible and clickable.

This brings the second point, Adsense ads do not and should not refresh. So if the existing ad units are refreshing on the page, this is an issue (for AdSense). If you ad new content to the page (sufficient content to warrant an additional ad-unit) then adding an ad is fine, but you should not refresh the existing ad (according to AdSense policies). If your content is changing, for example with an online tool or calculator and you need the ad to refresh, then you will need to serve your ads using DFP.

As for the Google crawling the content, this should be a concern not just for AdSense but for search as well. If you are adding sufficient new content to a page that would warrant an an additional ad unit then typically it would be desirable to have that content indexable by Google for Search. See my post in you other thread here: [webmasterworld.com...]
Ideally any new content should have its own URL that would allow users to link and share it and allow Google to index it. I would recommend using js history api specifically pushState() for that.

As for "Brand Safety" concerns, any new URL will need to be crawled and kept alive in the index. So if you are creating thousands of unique URL with personalized content then you should consider implementing canonical tags. If you are concerned that you are being impacted by "Brand Safety" you should check your coverage stats, specially for changes around December 15 2017.

[lucy24]

which is disallowed in the robots.txt

If in doubt, wouldn't it be wise to make a separate rule for the Adsensebot (I forget its formal name), allowing it to crawl selected scripts?