Are people blocking ads?

And if so, how to monetize the other impressions

[Kid_A]

A question for all. The disparity between my Adsense impressions and my internal server web logs from <my host> is growing to an amount I can longer ignore. AdSense has served over 150,000 impressions so far this month, but <my host> webstats say we've had over 350,000 page views.

I know SOME of those 350k+ page views are robots, but still... Is there any way to determine that? With spy blocking software, could it be that more and more people have javascript disabled, hence turning off the Google ads? Or maybe more people are intentionally using ad blockers?

Would a banner network be any different, in this case? And are there any that pay in the $1-$2 CPM? Last contact I had for banners was from Momma.com, and they were paying a laughable $0.25 CPM...

Thanks for your opinions,

[edited by: jatar_k at 4:27 am (utc) on Nov. 30, 2006]
[edit reason] no specific hosts thanks [/edit]

[MThiessen]

Google looks at unique IP (and other factors) to determine if it's the same person. The impressions on your server might have other factors as well.

If you have a robots.txt the ones in it that say "disallow" will STILL show in your stats, but google won't.

There are other factors as well, I believe google uses more then the IP to figure out if it is the same visitor, your logs won't though. They may look at things like:

Same IP range
Same Browser Version
Same screen resolution
equals same visitor.

If this same visitor clicks an ad you will likely get credit for it, it's just that the page impressions will look lower.

[gregbo]

I know SOME of those 350k+ page views are robots, but still... Is there any way to determine that?

Not perfectly, because fraudsters can easily fake the referrer, emulate legal bot behavior, etc.

[fredw]

But to answer your question...

I think the common wisdom here is that people who've blocked your ads won't click on an ad no matter how you force it on them.

That assumes, however, that your blocked users are blocked on purpose. I am increasingly concerned about the number of users who might have ads blocked but some other program made that decision for them, and if left up to the user, they'd be seeing and clicking on ads just fine...

Does anybody know if there's a way to discern how many blocked users are blocked on purpose as opposed to those who may be blocked involuntarily?

[Powdork]

What do you, and your host, consider a pageview.

For instance, my hosts stat program knows that images aren't pages, but it thinks my flash logo is a page. If you use frames (or Iframes), each time a user loads a page, several actual pages are loaded. I get thousands of calls for robots.txt per week and my hosts stat program counts those too. It would be cool if I could put adsense there, but i can't.

My recommendation is to get yourself a better stats program.

[gregbo]

Does anybody know if there's a way to discern how many blocked users are blocked on purpose as opposed to those who may be blocked involuntarily?

There's no reliable way to distinguish the two. Among other things, the criteria that you would use to determine this (e.g. a user agent that is suspected to block ads) can be faked.

[sunset_blvd]

I believe google uses more then the IP to figure out if it is the same visitor, your logs won't though. They may look at things like:

Same IP range
Same Browser Version
Same screen resolution
equals same visitor.

Actually, it goes a little something like this:

Google uses a unique cookie for each browser used by an individual user on a computer. This cookie contains information which allows Google to identify records about that user on its database. This cookie is submitted every time a user does a Google search, visits a site using AdSense etc. The information stored in Google's database, which is identified by the cookie, includes

* Everything you search for using Google
* Every web page you visit that has Google Adsense ads on it
* Every Blogger page you visit

If you have an Adsense account, the following will also be stored

* Your full name, address and bank account details
* The IP address of everyone who visits your pages with Adsense ads on them
* The number of visitors to each of your pages with Adsense ads on them

(excerpt from Wikipedia)

[Andreals]

Specifically the worry is the Auto-update of IE7. Does that beast reset the default in any way? I'm not seeing any recent reduction in impressions over traffic which suggests not but OTOH I should be seeing a Christmas spike in traffic given my topic.

[fredw]

I auto-updated on IE7 on one machine about a week ago, it did not change any settings pertaining to seeing Adsense ads.

[Andreals]

...it did not change any settings...

Good deal, the last thing I want is to be in the middle of a spat between Microsoft and Google.

edit: speling

[edited by: Andreals at 7:45 pm (utc) on Dec. 2, 2006]

[Kid_A]

OK: Here's a question. What if we put an otherwise invisible image at the bottom corner of our pages called "count-it"... would robots still crawl this image?

For that matter, wouldn't the loads of our logo, if I could track that, give us some indication of actual page loads?

But then I suppose some of those would be cached. And would robots not spider images anyway?

I just think that if our traffic is so much higher than what shows in Google - there are other ways to monetize that additional traffic. And if they are blocking intentionally - or not! - there are ways we can deal with that, too. For example, for those that are intentionally blocking ads, we could ask them to forward the site to friends. For those doing so unintentionally thanks to aggressive anti-spyware or whatever causes it... then we can find a new type of ad stream for them.

[Kid_A]

Actually:

I think I've figured it out. I checked out the OS types and Browsers... and I found we had 54k visits from the Yahoo Robot last month. It spiders through a ton of page and drives up the page view averages. Remove that, and our unique visitors match our Google numbers fairly closely. So I don't think I'm missing out on a ton of visitors.

It's not perfect but its close enough for my purposes.

[swa66]

also keep in mind

visitors with javascipt disabled

visitor hitting your html but clicking back, or a link of yours before the javascipt loads and gets an impression from Google

[fredw]

Because my site has features that would cause your online experience to be degraded if you have javascript turned off, I started monitoring the percentage of javascript to non-javascript users. I do this by including a pixel image in html and another presented by a simple javascript, and I have a report which compares their access stats.

My reports say consistantly that less than 1% of my users have javascript turned off. I'd consider that insignificant.

[swa66]

Don't forget that that 1% of your visitors are in fact the most web savvy visitors you'll have. I'm not that sure you should call them insignificant.

Anyway last time I measured I had more than 1%.

[netchicken1]

I use awstats and it doesn't include robots in the stats, they have their own table.

Are you looking at uniques comapared to all hits?

[iTunage]

If you wanted to measure the human pageviews you're getting, you'd need to cook up a bit of javascript in each page which links back to a server side file which keeps track of the count for you. Something like:
<script src="http://myserver.com/counter.php"></script>
It wouldn't output anything back to the user, but would keep track of actual browser visits as opposed to robots.
You could then use this count each month / week / day to compare with the impressions you're seeing through adsense. It won't be perfect, because as others have said, google has a much more precise measure of a pageview, but it will give you a much better comparison than your stats package does, because it will exclude everything that's not a person sitting at a browser.

[Powdork]

Don't forget that that 1% of your visitors are in fact the most web savvy visitors you'll have. I'm not that sure you should call them insignificant.

At least they think they are.

[Andreals]

People who turn off JavaScript are just ultra paranoid.

In the course of my business I surf hundreds of new sites weekly with Javascript on and have been doing this for years.

No ill affects that I'm aware of...

[inactivist]

Don't forget that that 1% of your visitors are in fact the most web savvy visitors you'll have. I'm not that sure you should call them insignificant.

Probably far less likely to click an ad too - in my experience, the more savvy the visitor, the less likely they are to click ads (of course, this could be a reflection of my site content, to be sure.)

Keep in mind that one does not have to disable javascript to block adsense - a simple hosts file entry will do the trick, and there are probably dozens of ad blocker programs that will kill adsense w/o disabling javascript. So, visitor stats that track javascript availability cannot tell the whole story.