|On premature conclusions with A/B testing|
| 1:30 pm on Oct 10, 2006 (gmt 0)|
I was getting some pretty amazing results from the officially endorsed A/B testing [adsense.blogspot.com] (CTR increased 6 times when using a new title colour).
Well, I was somewhat skeptical so I decided to test the reliablity of the A/B testing. I set up two channels A and B for a single ad unit. The important thing to note is that both channels used exactly the same colours, ad formats, etc. Hence for a reasonably high number of impressions the results for A and B should be very similar. But they were far from being similar.
Here are the results from the last Sunday:
CTR: 2 times higher than CTR of channel A,
eCPM: 9 times higher than eCPM of channel A
If I'd used a different color or format for one of the channels, I'd have made a wrong conclusion that one of the color palettes/formats performs significantly better than the other one.
[edited by: John_Carpenter at 1:51 pm (utc) on Oct. 10, 2006]
| 5:22 pm on Oct 10, 2006 (gmt 0)|
i'm trying to get a handle on this, but i guess that i need another cup of coffee.
both ads looked exactly the same, but one channel had significantly better earnings than the other channel?
what could be different? were both channels created at the same time? or is this some adsense fubar channel mess?
maybe keep it running for awhile, and see if the epc averages out?
| 5:44 pm on Oct 10, 2006 (gmt 0)|
This is really strange. Maybe historical channel data comes into affect and one of them was better targeted? Is that possible?
| 5:55 pm on Oct 10, 2006 (gmt 0)|
Interesting and useful experiment!
Surely the conclusion here is that, barring technical errors, the sample size (4000 impressions) was too small.
Would be interested to hear how large the sample size has to be to produce the expected results...
Update: you're switching the codes server-side, so maybe caching could be playing a role here?
E.g. let's say your page is cached by AOL's reverse proxy server, and all AOL users see only one of the ads... something like that.
| 7:22 pm on Oct 10, 2006 (gmt 0)|
Where the Ads showing on the content network?
If so, could it be that One Ad happened to appear on a particularly good site?
| 7:35 pm on Oct 10, 2006 (gmt 0)|
|you're switching the codes server-side, so maybe caching could be playing a role here? |
Well, I believe if that was the case, the number of impressions on channel A would have to differ significantly from channel B. But the number of impressions were:
Channel A: 4329
Channel B: 4273
The difference is about 1%, which I believe is not significant. As far as I can tell, both channels got practically the same exposure.
[edited by: John_Carpenter at 7:42 pm (utc) on Oct. 10, 2006]
| 7:40 pm on Oct 10, 2006 (gmt 0)|
|Where the Ads showing on the content network? |
Yes. Note that I'm only a publisher (I've never advertised via AdWords or anything else) and I'm talking about an experiment I carried out on our own site.