Welcome to WebmasterWorld Guest from 220.127.116.11
What if I have 10 conversions on 100 clicks over a month? Is that enough data to predict a 10% conversion rate going forward (in a static environment)?
Now 100 conversions on a thousand clicks over 2 months. Surely that is enough to predict a 10% conversion rate going forward with say 95% confidence?
In reality, I'll make decisions based on as few as 7 conversion but prefer more like 20.
What do you guys do?
so if i say yes, you can be 95% confident that some conversion ratios will hold constant, i'd be lying to you - they will change. i'm 100% confident of that. :-)
It is simple to mechanically apply formulas to data sets, but it's far harder to account for unexpected biases and new events that happen that introduce non-representative data into the population.
So go ahead and do it, but I wouldn't rely on it very much - the process you're applying it to is subject to many discontinuities that you can't predict.
It would be stable for quite a bit of time, and then suddenly we would question tracking because change would be too drastic. Tracking would be fine and we would stay in the “black”. Such “bumps” would not happen too often and for too long, but they would still happen.
In addition, depending on the type of product/service, there would be “seasons”. I say it under quotes because I refer to seasons that do not belong to those of Christmas type but unknown ones. I must admit that we have not been in this business long enough so we can recognize patterns that would help us fully understand these types of changes. Two or three years are just the beginning of the period when you can compare month to month, quarter to quarter and so on. I guess 5 years is good enough…
… but then, again, something that has never happened before (like what RhinoFish says) can kick in and make you feel “no brain”.
I would somewhat rely on that data and watch closely. Any huge plan (like some kind of investment) would need a careful examination.
Also, please keep in mind that data in your example looks “artificial”: 100 clicks in one month and 1000 clicks in two months while it should be 200 clicks in two months.
If you are referring to keywords, and think about finding more of them over the time, each keyword will have its own conversion. If you think about expanding onto new PPC areas, each of them has its own conversion, again.
Maybe this could be used here: If you go to the bank and ask about mutual funds, nobody will guarantee you anything. But, when you look into graphs, most of them look OK on the long run, right?
Sometimes it’s how much you believe into something. Did you watch “Polar Express”? You believe?
Statistical analysis is the only thing that you can count on... if you're in it for the long haul. Unpredictable events happen, but they go equally good and bad. In the end, the anomolies balance themselves out and the statistics are still valid.
Did you look up the 6 sigma stuff? It accounts for those anomolies and also gives you guidelines for identifying the bumps (heaviside functions for the math geeks). This isn't new theory, it's old, tried and proven.
Embrace the math. Use statistical analyis. You can identify the cycles, the trends and set alarms for the anomolies. If you do anything else it's called gambling and all those gamblers are what make Vegas rich. Me, I'd rather be the casino... or a bank, or a stock broker, or an insurance company... Which mega company doesn't use stats as a fundamental tool for running their company?
Online advertising gives you an absolutely insane amount of data. If you don't rely on it you're crazy. You can't get that much data and fail unless you ignore it. Do you really think you have a formula that can beat the odds? Luckily, 95% of advertisers out there chase red herrings and live in fear of being blind-sided. 95% of advertisers don't do the math. Lucky us who do!
One_Thing_Well is asking the right question. If he follows his gut and pursues the answers he'll be doing very, very well.
We continue to enter new markets and rock each one because we do the math. We've done it over 20 times! Hopefully One_Thing_Well will take this advice and run with it. In a couple weeks this post will be buried and only the hand full of people who read it will get such clear direction. Of course, 95% of them will ignore it and that's good for me :)
My post came across to you wrong, I wrote it with a smile, not a scowl. :-)
"Statistical analysis is the only thing that you can count on... if you're in it for the long haul."
I didn't say don't use it. In fact, I said to use them but to not rely on it too heavily. I use lots of analytical techniques myself. But it's been my experience that when people start asking about confidence levels based on standard devs, they will apply things with far too high a degree of confidence themselves.
"Did you look up the 6 sigma stuff? It accounts for those anomolies and also gives you guidelines for identifying the bumps (heaviside functions for the math geeks). This isn't new theory, it's old, tried and proven."
I'm well aware of the practice and of the math. I still assert that you need a highly capable process to be under study and that what we do doesn't qualify. I have enough empirical evidence to know that. Again, that doesn't mean it's useless to me to apply mean and standard dev techniques to the numbers, but they must be viewed with an ongoing skepticism. The process analyzed needs to follow a Gaussian distribution (though substitutions of other known process / dist curves can be studied as well) for the studies to be rigorously meaningful. I argue that they are not, however, using the statiscal analysis can still be worthwhile for trend analysis and ppc control / feedback, but ignoring the shortcomings of this method will leave you disappointed, and at times, surprised.
Even the name six sigma that you've used is not applicable to what we do for a living. At six sigma, in a tightly controlled manufacturing process, you'll get 3.4 bad products per million produced. In our world, that would mean you're able to predict and achieve sales outcomes with a degree of accuracy that is simply not attainable.
"Embrace the math. Use statistical analyis. You can identify the cycles, the trends and set alarms for the anomolies. If you do anything else it's called gambling and all those gamblers are what make Vegas rich. Me, I'd rather be the casino... or a bank, or a stock broker, or an insurance company... Which mega company doesn't use stats as a fundamental tool for running their company?"
Again, I didn't say don't analyze trends - I said calculating degrees of confidence will likely lead you to be overly confident and unable to see the unexpected coming.
If you choose to use less rigorously imposing analysis than six sigma, I wouldn't call that gambling. Tens of thousands of businesses in the US do sales forecasts without degree of confidence calculations, using many other statistical analysis methods. This doesn't make them all gamblers.
"We continue to enter new markets and rock each one because we do the math. We've done it over 20 times! Hopefully One_Thing_Well will take this advice and run with it. In a couple weeks this post will be buried and only the hand full of people who read it will get such clear direction. Of course, 95% of them will ignore it and that's good for me :)"
We don't differ as much as you may think, both in results and application of statistical trend analysis. It's more that I don't view it something that would make a booming success of everyone who would read and follow your SA suggestions here. There are other very tangible skills you are using that you aren't giving enough credit. And the marketing world's best success stories aren't centered on mathmaticians or their sigmas.
So rock on! And I wish you zero defects and no negative outliers! :-)
Because of that it's important to continue testing and monitoring everything so you can stay ahead of search trends. If you don't make changes, you'll continue to see declines of click through rates and all other metrics pretty steadily over time.
Here's the rule of thumb:
12 conversions = strongly confident
7 conversions = reasonable confidence
3 conversions = usable, but low confidence
BTW, there's no magic about that 95% confidence level. That's just the level you need to get published in social science journals, which is why stats professors end ups mentioning it so much.
Something with more options than split tester? I did a google search but the results make my head spin.