homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google AdSense
Forum Library, Charter, Moderators: incrediBILL & jatar k & martinibuster

Google AdSense Forum

On premature conclusions with A/B testing
John Carpenter

 1:30 pm on Oct 10, 2006 (gmt 0)

I was getting some pretty amazing results from the officially endorsed A/B testing [adsense.blogspot.com] (CTR increased 6 times when using a new title colour).

Well, I was somewhat skeptical so I decided to test the reliablity of the A/B testing. I set up two channels A and B for a single ad unit. The important thing to note is that both channels used exactly the same colours, ad formats, etc. Hence for a reasonably high number of impressions the results for A and B should be very similar. But they were far from being similar.

Here are the results from the last Sunday:

Channel B:
Impressions: 4,273
CTR: 2 times higher than CTR of channel A,
eCPM: 9 times higher than eCPM of channel A

As a side note, I did not use the official A/B testing JavaScript code, but rotated the ad codes randomly using server-side scripting (PHP). However, this should not play any role (I've tested the random number generator and didn't find any significant problem).

If I'd used a different color or format for one of the channels, I'd have made a wrong conclusion that one of the color palettes/formats performs significantly better than the other one.

[edited by: John_Carpenter at 1:51 pm (utc) on Oct. 10, 2006]



 5:22 pm on Oct 10, 2006 (gmt 0)

i'm trying to get a handle on this, but i guess that i need another cup of coffee.

both ads looked exactly the same, but one channel had significantly better earnings than the other channel?

what could be different? were both channels created at the same time? or is this some adsense fubar channel mess?

maybe keep it running for awhile, and see if the epc averages out?


 5:44 pm on Oct 10, 2006 (gmt 0)

This is really strange. Maybe historical channel data comes into affect and one of them was better targeted? Is that possible?


 5:55 pm on Oct 10, 2006 (gmt 0)

Interesting and useful experiment!

Surely the conclusion here is that, barring technical errors, the sample size (4000 impressions) was too small.

Would be interested to hear how large the sample size has to be to produce the expected results...

best, a.

Update: you're switching the codes server-side, so maybe caching could be playing a role here?

E.g. let's say your page is cached by AOL's reverse proxy server, and all AOL users see only one of the ads... something like that.

What happens if you switch to the Javascript, client-side A/B test?


 7:22 pm on Oct 10, 2006 (gmt 0)

Where the Ads showing on the content network?

If so, could it be that One Ad happened to appear on a particularly good site?

John Carpenter

 7:35 pm on Oct 10, 2006 (gmt 0)

you're switching the codes server-side, so maybe caching could be playing a role here?

Well, I believe if that was the case, the number of impressions on channel A would have to differ significantly from channel B. But the number of impressions were:

Channel A: 4329
Channel B: 4273

The difference is about 1%, which I believe is not significant. As far as I can tell, both channels got practically the same exposure.

[edited by: John_Carpenter at 7:42 pm (utc) on Oct. 10, 2006]

John Carpenter

 7:40 pm on Oct 10, 2006 (gmt 0)

Where the Ads showing on the content network?

Yes. Note that I'm only a publisher (I've never advertised via AdWords or anything else) and I'm talking about an experiment I carried out on our own site.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google AdSense
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved