How many impressions are needed to effectively

test a group of ads?

[mona]

I searched around a bit and couldn't find much about this topic. I know there's no clear cut answer, but is there a ball park figure that seems to work? I can see three factors to take into account. The number of impressions, the number of ads, and how big the gap is in the CTR number. Am I missing any? (Besides conversion rate, see below)

If I'm running 3 ads with 500 impressions and the CTRs are
4.7
3.1
.09
I think it would be safe to drop the .09 ad. But what if you have 500 impressions with 3 ads and the CTRs are
2.7
2.2
1.9
How many more impression are needed? Thanks.

There's a delay in setting up the conversion tracking, so I can't take that into account right now

[sullen]

For CTR, I'd say 500 is a reasonable number to base decisions on.

However, I suggest you do wait for the conversion tracking, and wait for 500 at least 5000 impressions before making a decision. Sometimes a lower CTR gives a better conversion.

[eWhisper]

There have been a lot of threads about statistically significant ads [google.com] on this issue.

One point that always sticks in my mind from the DM world - it's 7 actions that define a test, not CTR or impressions.