I suppose I have the following results:
_______Clicks___Impressions___Click-Through-Rate
Ad 1___6________330___________1.81%
Ad 2___5________337___________1.48%
Ad 3___4________326___________1.22%
Total__15_______993
Now, if you just look at it, you can say Ad 1 is the best because you get highest click-through-rate.
My concern is that, the ranking of the the results as shown above could be just a random user behavior.
So my theory is that if I have more impressions (bigger sample size), then I should be able to make better decision. So my question is that should I wait until 2000 impressions? 3000? 4000? what should be the impressions size? does it even matter?
Thanks!
Paul
here's an article about the math behind determining the minimum sample size:
[isixsigma.com...]
and here's the skinny for you... the size of the dataset required isn't an absolute answer. the question isn't "how big a set do I need", it's "how big a set do i need to reliably forecast with a certain degree of confidence"... so you see, people here can't give you an absolute answer, because we know everyone is looking for varying amounts of uncertainty in their forecasts. conservative people want a high degree of certainty, while others will accept more risk. further, also inherent in this question, is the margin of error and repeatability of the outcomes - this varies depending on many factors including your ad texts, your landing page, your competitors, your bids, some consumer behaviors, your dayparting, your channels of distribution, and much more. lastly, the calculation for sufficient dataset size also assume a steadt state process, sometimes called one "under control", but your competitors are changing things all the time in this auction, so there's a lot of changing variables at work here that make this a process that's not under control.
as you progress, you'll get a general feel for what a sufficient sample size is, for the way (or ways) that you do things. i suggest starting with a large number and working your way down as you learn that the things you change based on judgments using your learned "thumbrule" are making an actual performance difference.
right now, you've asked us if 6 clicks is enough to make a decision, that answer is definitely no. at 500 clicks, I'd say you'd have a very reasonable level of confidence (though still undefined due to ever changing nature of the auction settind) to make sound decisions.
so question is, where between 6 and 500 is a faster, nearly-as-reliable place? it's really for you to decide. i'd say if you made tuning decisions at 100 or less, you'd generally be a "wing it" kind of person, and if you make them at 300 or so, you're a "middle of the roader", and if you wait for 500 or more, you're pretty conservative and averse to risk.
note that I said tuning decisions... optimizing requires a lot of data. however, rough adjust decisions require less data. if i were planning on running 300 clicks towards optimizing a new project, and was expecting a 2% conversion rate and i'd reached 150 clicks and had no sales yet (on average, 2% conversion gives me 3 sales at 150 clicks), i'd likely stop the test and declare the effort a failure and go back to the drawing board. but again, these numbers reflect my biases and what i consider to be an acceptable level of risk, your number should and will vary.
