Welcome to WebmasterWorld Guest from

Forum Moderators: open

Minimum sample size for testing.

any math / stats whizzes out there?

5:38 pm on Jan 31, 2006 (gmt 0)

10+ Year Member


Here's the basic idea:

I'm running several PPC ad campaigns and trying to determine the optimum bid price.

My question is this: what sample size must I take to make sure that the numbers are statistically significant?

Here's an example:

6000 impressions
52 clicks ( gives a .8% CTR roughly)
2 conversions (3.85%)

Now,I'm not really concerned about the CTR, but the actual conversion ratio.

Most of are successful campaigns are seeing around 9-10% conversion. I want to be able to test for 1-2 weeks, and know with 95% certainty that the conversion rate I see will be within say, 1% (maybe .5%) of the results. How many samples (clicks) must I record to find this?


4:16 am on Feb 8, 2006 (gmt 0)

10+ Year Member

If you run what is known as a monte carlo analysis
(statistical simulation) 200 samples is usually enough.
However, in the case of marketing there can be delays
from click to conversion. I have found that after 1000 clicks statistics start to converge to a realistic number. In some cases I have noticed a good conversion rate for new ads and then after running the ad for a while it slows down.

2:23 pm on Feb 9, 2006 (gmt 0)

10+ Year Member

1000 clicks?

I have about 450 keywords in my Overture account, some bids are around 50p per click. You mean I'd have to spend 500 on 1 keyword before seeing whether it's converting or a waste of money? Wouldnt 100 clicks or less be enough?


2:22 pm on Feb 12, 2006 (gmt 0)

5+ Year Member

A good rule of thumb in any statistical process is that the 'noise' will be plus/minus the square root of the total events.

To get within 1%, you need root(n)/n = 1/100.
So n = 10,000 conversions.

You simply don't have the numbers to get that sort of statistical confidence within the time frame.

9:45 pm on Feb 17, 2006 (gmt 0)

5+ Year Member

Seasonality play a role so when you run a campaign is also something to consider (unless it's not a seasonal product). I think it's probably safe to assume that you'd get far fewer conversions two months away from Christmas as opposed to two weeks away.

Featured Threads

Hot Threads This Week

Hot Threads This Month