I'm generally familiar with the concepts behind A/B vs MVT testing and measuring for statistical significance when it comes to testing pages, but would somebody please explain how to take the revenue into consideration for testing a shopping cart variations?
For instance, if I'm testing shopping cart variations with various cross sells, or complementary products, and the test yielded in relatively consistent conversion rates, how do I know if my data is statistically significant?
If I have 10 visits to each cart variation and 1 transaction, this is certainly not enough data to make a statistically significant conclusion.
What about the following scenario:
Cart 1 - 10k visits, 1000 conversions, $5000 revenue
Cart 2 - 10k visits, 1000 conversions, $6000 revenue
Cart 3 - 10k visits, 1000 conversions, $7,000 revenue
Surely this must be enough data to choose cart variation 3, but how is that calculation made?
How about the following extreme scenario where I'm testing the price of a poster in the cart but have very little traffic.
Cart 1 - 20 visits, 10 conversions, $100 revenue ($10 product)
Cart 2 - 20 visits, 8 conversions, $800k revenue ($100k product)
A split test calculator tells me I need 200+ visitors to reach 90% statistical significance for determining which cart converts better. Intuition says to keep selling the magazine at $100k, as there were actually 8 people willing to buy the poster for $100k - there are probably more. Can you make a judgement call to permanently sell it at $100k even though the data indicates the test is inconclusive?
Any advice or pointers to a book or site would be very appreciated. I'm wishing I had paid more attention in my statistics class...