Minimum sample size for testing.

any math / stats whizzes out there?

[nquinn]

So,

Here's the basic idea:

I'm running several PPC ad campaigns and trying to determine the optimum bid price.

My question is this: what sample size must I take to make sure that the numbers are statistically significant?

Here's an example:

6000 impressions
52 clicks ( gives a .8% CTR roughly)
2 conversions (3.85%)

Now,I'm not really concerned about the CTR, but the actual conversion ratio.

Most of are successful campaigns are seeing around 9-10% conversion. I want to be able to test for 1-2 weeks, and know with 95% certainty that the conversion rate I see will be within say, 1% (maybe .5%) of the results. How many samples (clicks) must I record to find this?

Thanks!

[TheGuyAboveYou]

If you run what is known as a monte carlo analysis
(statistical simulation) 200 samples is usually enough.
However, in the case of marketing there can be delays
from click to conversion. I have found that after 1000 clicks statistics start to converge to a realistic number. In some cases I have noticed a good conversion rate for new ads and then after running the ad for a while it slows down.

[mrclark]

1000 clicks?

I have about 450 keywords in my Overture account, some bids are around 50p per click. You mean I'd have to spend £500 on 1 keyword before seeing whether it's converting or a waste of money? Wouldnt 100 clicks or less be enough?

Steve

[Scruffy]

A good rule of thumb in any statistical process is that the 'noise' will be plus/minus the square root of the total events.

To get within 1%, you need root(n)/n = 1/100.
So n = 10,000 conversions.

You simply don't have the numbers to get that sort of statistical confidence within the time frame.

[wsm2006]

Seasonality play a role so when you run a campaign is also something to consider (unless it's not a seasonal product). I think it's probably safe to assume that you'd get far fewer conversions two months away from Christmas as opposed to two weeks away.