I was getting some pretty amazing results from the officially endorsed A/B testing [adsense.blogspot.com] (CTR increased 6 times when using a new title colour).
Well, I was somewhat skeptical so I decided to test the reliablity of the A/B testing. I set up two channels A and B for a single ad unit. The important thing to note is that both channels used exactly the same colours, ad formats, etc. Hence for a reasonably high number of impressions the results for A and B should be very similar. But they were far from being similar.
Here are the results from the last Sunday:
CTR: 2 times higher than CTR of channel A,
eCPM: 9 times higher than eCPM of channel A
If I'd used a different color or format for one of the channels, I'd have made a wrong conclusion that one of the color palettes/formats performs significantly better than the other one.
[edited by: John_Carpenter at 1:51 pm (utc) on Oct. 10, 2006]