Analysis of Adsense Channel Statistics

when is CTR data statistically significant

[elsewhen]

i am doing a lot of data collection with adsense channels, and the information is very interesting. right now, i am doing a test regarding ad colors.

due to my poor understanding of statistics, i thought that i needed to wait until i got 30 clicks until the CTR data is statistically significant.

i decided to wait, and now i have reached over 130 clicks in each channel and the data is still shifting slightly. when can i stop the test and be confident that the data is not arbitrary.

by the way... the various color schemes are randomly assigned to pages with a server-side script.

this type of testing costs money because i have some poorer-performing colors in the mix; i would to stop it as soon as i have valid data.

thanks for any help with this.

[JohnKelly]

I did something similar, and used three days of data (several hundred clicks in the test case). I don't thing making assumptions based on intraday data is a good idea.

[rbacal]

OK. A few things. The major problem isn't statistical signficance. You need to make sure that your "experiment" is set up to determine whether there is a cause-effect relationship between two things (eg. color, CTR). That means that you need to create a situation where every other possible variable is "held constant" (you can get around that with some very sophisticated statistical techniques, but trust me, you won't want to, and probably can't without advanced learning.

So, while you may have "enough" of a sample to satisfy the math of it, you need a much longer period to satisfy the logic to rule out other causes.

Let's say you take data from Monday and Tuesday and Wednesday (which gives you a good size sample). You draw a conclusion from that large sample that changing colors yields higher ctr. But what happens if there is something different about the people who visit your site on those days that affects CTR? You won't know that because you don't have data from other days. Maybe people who visit during the week tend to like green, but Friday people like red?

So, yes, you need "enough" data, but you need the right data (properly sampled which means it needs to be done over time.

Aside: your sample size isn't number of clicks. The number of clicks in your sample isn't what you should be looking at. Probably ad impressions or page impressions would be better. How I would do it is consider each day's total an observation (a single data point). I used to teach research and statistics but admit to being a bit rusty on this point, but I'd go with days.

With this kind of thing, you can cut corners, use smaller samples, less data, etc, but the more you do that the more likely you are to draw an incorrect conclusion, and never realize it.

[arrowman]

due to my poor understanding of statistics, i thought that i needed to wait until i got 30 clicks until the CTR data is statistically significant.

No, no, no! You must wait for 42 clicks.