Forum Moderators: martinibuster
My question is how many impressions/ or days (in terms of time) should I wait to get the idea if new design is working or not.
I also belive that you need hundreds if not thoushands of clicks to start see the effects of many changes you may be making.
how many impressions/ or days (in terms of time) should I wait
It depends on how much variation is normally in the data being measured. For example, if the goal is increased clicks, then calculate the moving average of clicks per day, starting with a window of one day, and expanding the window until the resulting curve looks "fairly smooth". If that takes, for example, a window of 30 days, then you'll want to give your changes 30 days before comparing.
Or, looking at it the other way, pick the # of days you want to perform your test, calculate the past variability in your data for that # of days, and then you know a big a change your test would have to produce in that amount of time to be deemed significant.
For example, you decide you only want to do a 3-day test. So, take the last 90 days, and average days 1,2,3 to make your first data point, so on until you have 30 data points. Plug those into a spreadsheet or calculator and ask for the standard deviation, then double it. Suppose you discover that your average clicks per day for the past 90 days was 400, and that twice the standard deviation of the past 30 3-day periods happens to be equal to 250. That tells you that your that for your test to give a clear signal of being an improvement, you would really want to see it produce an average of 650 clicks per day or better.
Now, if you know your average has been 400 clicks per day, and you see that your test produced 525 clicks per day, you're going to have a really strong feeling that your test was a success. But you'll just be fooling yourself, since there'll be about a 30% chance that you could have got the same results with your old website before the changes being tested were applied. That's why you need to run the numbers -- "gut feel" is really bad at detecting what's significant and what's not.
You have to look at the variance in your data to make an intelligent choice about how long a test to run. The correct answer for you depends on your data, and any number anybody could give you without first inspecting the amount of variance in your data is useless.
