Welcome to WebmasterWorld Guest from 54.161.246.212 register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts, Pubcon Platinum Sponsor 2014
 Home / Forums Index / WebmasterWorld / Ecommerce
Forum Library, Charter, Moderators: buckworks

# Ecommerce Forum

 Tweet
Shopping Cart Multivariate Testing - Statistical Signficance
parke02

Msg#: 4262008 posted 1:55 am on Feb 3, 2011 (gmt 0)

Hi All,

I'm generally familiar with the concepts behind A/B vs MVT testing and measuring for statistical significance when it comes to testing pages, but would somebody please explain how to take the revenue into consideration for testing a shopping cart variations?

For instance, if I'm testing shopping cart variations with various cross sells, or complementary products, and the test yielded in relatively consistent conversion rates, how do I know if my data is statistically significant?

If I have 10 visits to each cart variation and 1 transaction, this is certainly not enough data to make a statistically significant conclusion.

Cart 1 - 10k visits, 1000 conversions, \$5000 revenue
Cart 2 - 10k visits, 1000 conversions, \$6000 revenue
Cart 3 - 10k visits, 1000 conversions, \$7,000 revenue

Surely this must be enough data to choose cart variation 3, but how is that calculation made?

How about the following extreme scenario where I'm testing the price of a poster in the cart but have very little traffic.

Cart 1 - 20 visits, 10 conversions, \$100 revenue (\$10 product)
Cart 2 - 20 visits, 8 conversions, \$800k revenue (\$100k product)

A split test calculator tells me I need 200+ visitors to reach 90% statistical significance for determining which cart converts better. Intuition says to keep selling the magazine at \$100k, as there were actually 8 people willing to buy the poster for \$100k - there are probably more. Can you make a judgement call to permanently sell it at \$100k even though the data indicates the test is inconclusive?

Any advice or pointers to a book or site would be very appreciated. I'm wishing I had paid more attention in my statistics class...

Thanks,

-Eugene

parke02

Msg#: 4262008 posted 8:12 pm on Mar 2, 2011 (gmt 0)

bump

RhinoFish

Msg#: 4262008 posted 2:27 pm on Mar 3, 2011 (gmt 0)

> Cart 1 - 20 visits, 10 conversions, \$100 revenue (\$10 product)
> Cart 2 - 20 visits, 8 conversions, \$800k revenue (\$100k product)

did you mean to have the "k" in the second one, and not in the first...

you have a 60% conversion rate of a \$100,000 product?

p5gal5

Msg#: 4262008 posted 10:37 pm on Mar 3, 2011 (gmt 0)

To acquire a statistically significant result, you will need one either:
1. A greater difference in the outcomes
2. An extremely large dataset
3. Less variance within each group (site)

90% statistical significance is not very high - usually 95 or 99 are considered benchmarks.

As variance is a determining factor in statistical significance, if your transaction values vary (you carry more than one product at one price), it will take longer (more visitors) to achieve a meaningful result. There are lots of books on this, but they are textbooks (and I'd recommend some advanced calculus before tackling them).

 Global Options: top home search open messages active posts  Tweet

 Home / Forums Index / WebmasterWorld / Ecommerce