Evaluating a split A/B test on eCPM

A statistical question

[frox]

Im an running an A/B test, and trying to evaluate its results.

I have found several online calculators that allow me to assess the significance of the different CTRs I get.

For example,

channel A: 100000 impressions, 900 clicks
channel B: 100000 impressions, 1000 clicks

One site (can't post here the URL) tells me that there is 0.02 probability that it's a mere-chance difference. (which I read as being equivalent to a relevance of 0.98 ... Am I wrong?)

The problem is ... that kind of formula is ok if we speak about CTR (a percentage of the number of samples) but what is the formula to use to evaluate differences in eCPM?

If I do A/B test on color, for example, there is no reason that I should have a different eCPM. On the other hand, if the A/B test is on ads format, a larger format might contain more ads and so my eCPM might be different. Even more, if the A/B test involves for example replacing an ads unit with a links unit, comparing clicks would be useless, I need to compare earnings (or eCPM)

So, if in the same scenario I have
channel A: 100000 impressions, $90
channel B: 100000 impressions, $100
What is the statistical relevance of this data?
Is there a formula to calculate this?

(I have been trying to understand if the same calculator could be used in this case, but evidently not.)

thanks for any help!

[btas2]

You don't have enough info to tell.

eCPM depends on a bunch of hidden variables (like smart pricing) and the earnings for a given click on a given ad may be different on different days and even at different times on the same day.

That's what makes AdSense so difficult to understand, and I suspect that it's not by accident either.

You can tell which of two layouts gets the most clicks (though even there it depends somewhat on which ads Adsense decides to serve), but it's really difficult to say which of two layouts will generate the highest earnings on a consistant basis.

I can't even predict what my earnings will be day to day with the same layout! If you gather enough data to actually make some sort of prediction, AdSense may well do an update and all the data will change overnight.

[bcc1234]

Most likely, you are talking about either fisher's test or chi square. If that's the case, then no, you can't use a dollar amount and plug it into the formula directly. Your best bet is to stick with counting clicks for all attributes you are testing.