Welcome to WebmasterWorld Guest from 188.8.131.52
Well, I was somewhat skeptical so I decided to test the reliablity of the A/B testing. I set up two channels A and B for a single ad unit. The important thing to note is that both channels used exactly the same colours, ad formats, etc. Hence for a reasonably high number of impressions the results for A and B should be very similar. But they were far from being similar.
Here are the results from the last Sunday:
CTR: 2 times higher than CTR of channel A,
eCPM: 9 times higher than eCPM of channel A
If I'd used a different color or format for one of the channels, I'd have made a wrong conclusion that one of the color palettes/formats performs significantly better than the other one.
[edited by: John_Carpenter at 1:51 pm (utc) on Oct. 10, 2006]
both ads looked exactly the same, but one channel had significantly better earnings than the other channel?
what could be different? were both channels created at the same time? or is this some adsense fubar channel mess?
maybe keep it running for awhile, and see if the epc averages out?
Surely the conclusion here is that, barring technical errors, the sample size (4000 impressions) was too small.
Would be interested to hear how large the sample size has to be to produce the expected results...
Update: you're switching the codes server-side, so maybe caching could be playing a role here?
E.g. let's say your page is cached by AOL's reverse proxy server, and all AOL users see only one of the ads... something like that.
you're switching the codes server-side, so maybe caching could be playing a role here?
Channel A: 4329
Channel B: 4273
The difference is about 1%, which I believe is not significant. As far as I can tell, both channels got practically the same exposure.
[edited by: John_Carpenter at 7:42 pm (utc) on Oct. 10, 2006]