| Shopping Cart Multivariate Testing - Statistical Signficance
|
parke02
msg:4262010 | 1:55 am on Feb 3, 2011 (gmt 0) |
Hi All,
I'm generally familiar with the concepts behind A/B vs MVT testing and measuring for statistical significance when it comes to testing pages, but would somebody please explain how to take the revenue into consideration for testing a shopping cart variations?
For instance, if I'm testing shopping cart variations with various cross sells, or complementary products, and the test yielded in relatively consistent conversion rates, how do I know if my data is statistically significant?
If I have 10 visits to each cart variation and 1 transaction, this is certainly not enough data to make a statistically significant conclusion.
What about the following scenario:
Cart 1 - 10k visits, 1000 conversions, $5000 revenue
Cart 2 - 10k visits, 1000 conversions, $6000 revenue
Cart 3 - 10k visits, 1000 conversions, $7,000 revenue
Surely this must be enough data to choose cart variation 3, but how is that calculation made?
How about the following extreme scenario where I'm testing the price of a poster in the cart but have very little traffic.
Cart 1 - 20 visits, 10 conversions, $100 revenue ($10 product)
Cart 2 - 20 visits, 8 conversions, $800k revenue ($100k product)
A split test calculator tells me I need 200+ visitors to reach 90% statistical significance for determining which cart converts better. Intuition says to keep selling the magazine at $100k, as there were actually 8 people willing to buy the poster for $100k - there are probably more. Can you make a judgement call to permanently sell it at $100k even though the data indicates the test is inconclusive?
Any advice or pointers to a book or site would be very appreciated. I'm wishing I had paid more attention in my statistics class...
Thanks,
-Eugene
|
parke02
msg:4275468 | 8:12 pm on Mar 2, 2011 (gmt 0) |
bump
|
RhinoFish
msg:4275908 | 2:27 pm on Mar 3, 2011 (gmt 0) |
> Cart 1 - 20 visits, 10 conversions, $100 revenue ($10 product)
> Cart 2 - 20 visits, 8 conversions, $800k revenue ($100k product)
did you mean to have the "k" in the second one, and not in the first...
you have a 60% conversion rate of a $100,000 product?
|
p5gal5
msg:4276272 | 10:37 pm on Mar 3, 2011 (gmt 0) |
To acquire a statistically significant result, you will need one either:
1. A greater difference in the outcomes
2. An extremely large dataset
3. Less variance within each group (site)
90% statistical significance is not very high - usually 95 or 99 are considered benchmarks.
As variance is a determining factor in statistical significance, if your transaction values vary (you carry more than one product at one price), it will take longer (more visitors) to achieve a meaningful result. There are lots of books on this, but they are textbooks (and I'd recommend some advanced calculus before tackling them).
|
|
|
