Adsense Ads & 404 Issue

[gouri]

I was looking at my log files and I noticed some 404 messages which I believe are coming from pages that have adsense ads on them.

I then did a search on Webmasterworld and found an old thread that was relevant to what I am seeing:

[webmasterworld.com...]

I looked up some of the ip addresses that came up in my log and I see it that it could be from some company called Hughes. The thread above also makes mention of this.

Can someone please explain to me why a page that has adsense ads on it may be giving a 404 message?

Here is another thread that is related to the issue:

[webmasterworld.com...]

I don't know if this is related but I ran XenuSleuth and for the adsense address

http://pagead2.googlesyndication.com/pagead/show_ads.js

I am receiving the status as no connection and the error message A connection to the server could not be established

I would appreciate your help.

[edited by: incrediBILL at 6:53 pm (utc) on Dec. 23, 2009]
[edit reason] unlinked AdSense URL [/edit]

[Lame_Wolf]

I am pleased someone has brought this up again. I still see it in my logs, but never see it when I visit the site.

However... if you Google "/pagead/render_ads.js" you will see sites that actually render the code so a human can see it when they visit the page. But never on my site.

[gouri]

I googled it and I found websites in the SERP that seem to have /pagead/render_ads.js (for example [pagead2.googlesyndication.com...] on the page as regular text just like you would see a sentence on soccer if you went to a soccer website.

Now I am trying to understand why this would appear as regular text on a webpage that isn't yours and also appear in someone's server logs for their own websites? I don't see how these two are related.

[sid786]

Here is another that was started by me [webmasterworld.com...]

[levo]

and another [webmasterworld.com...]

[Lame_Wolf]

ASA ?

[caribguy]

Looks like satellite internet: pre-fetching gone wrong. 148.* does all kinds of "interesting" things.

67.142.0.0/15
72.168.0.0/14
148.62.0.0/15
148.64.0.0/13
148.72.0.0/14
148.76.0.0/15
148.78.0.0/16

Some common entries:

"GET /google-analytics.com/ga.js HTTP/1.0" 404 (Analytics)
"GET /maps/c/ui/HovercardLauncher/dommanifest.js HTTP/1.0" 404 (Google Maps)
"GET /XdComm.swf HTTP/1.0" 404 (Facebook connect)
"GET /a123bcde.gif HTTP/1.0" 404 (Facebook "fan" images)
"GET /pagead/osd.js HTTP/1.1" 404 (Note HTTP/1.1)

Edit: added ranges

[bcc1234]

I'm seeing a lot of 404s for:

/pagead/test_domain.js
/pagead/osd.js
/pagead/expansion_embed.js
/pagead/render_ads.js

It looks like sometimes javascript isn't executed properly and the host name is not set, so the client tries to fetch those files from the same host that served the page.

I found one discussion about it and the advice was to set up a redirect on your host for those file names, and redirect them to pagead2.googlesyndication.com and/or doubleclick's host.

I'm not sure if that would help or if that would even be allowed by TOS.

It would be nice if ASA could provide us with a definitive answer on this issue.

[Lame_Wolf]

I found one discussion about it and the advice was to set up a redirect on your host for those file names, and redirect them to pagead2.googlesyndication.com and/or doubleclick's host.

Even if theis were true, we shouldn't be doing this. It's Google's problem to fix, not ours.

[levo]

and another one /doubleclick/DARTIframe.html

[KenB]

I think some of these odd 404 errors we see are the result of an ISP's prefetching engine incorrectly parsing JavaScript files. What happens is that some ISPs, especially Satellite based ISPs, try to accelerate page loading for users by having caching servers fetch images, CSS files, JavaScripts, etc. while the parent HTML file is in the process of being transferred to the user. Sometimes, however they improperly parse JavaScript files and thus cause a lot of 404 errors.

See [webmasterworld.com...] for a discussion on this.

[gouri]

@KenB

What you wrote is very informative.

I was wondering, if the number of 404s on a website is increasing by the same amount everyday (by the number of pages on the website that have Adsense) and it is from the same country, I don't think it is from individuals trying to view the website. I think it is some sort of robot or something.

Can you please think of what might be happening from the ISP end to cause this or is it possibly something else? What could be causing this to occur everyday?

[gouri]

Can you please think of what might be happening from the ISP end to cause this or is it possibly something else? What could be causing this to occur everyday?

I didn't mean only KenB when I wrote this, I meant if anyone might know something about it.

[KenB]

In the case I was discussing in the thread I linked to above, what is happening is that that the acceleration engine the ISP is using is trying to parse JavaScripts, but is failing to properly concatenate URL strings in the JavaScripts. For instance if they see the following example code, they will pick up on the "/example.gif" but miss the "http://example.com" part of the concatenation.

var a="http://example.com";
return a+"/example.gif";

In my case, my logs are filling with 404 errors caused by this issue from the same ISP. Apparently they have a really large user base (approx 1,000,000 subscribers). As I said in the other thread they are aware of the issue and are working to fix it. Hopefully we will see a fix in the next month or two.