Welcome to WebmasterWorld Guest from 54.196.125.165

Forum Moderators: buckworks

Message Too Old, No Replies

Shopping Cart Multivariate Testing - Statistical Signficance

parke02

1:55 am on Feb 3, 2011 (gmt 0)

New User

joined:Feb 3, 2011
posts:2

Hi All,

I'm generally familiar with the concepts behind A/B vs MVT testing and measuring for statistical significance when it comes to testing pages, but would somebody please explain how to take the revenue into consideration for testing a shopping cart variations?

For instance, if I'm testing shopping cart variations with various cross sells, or complementary products, and the test yielded in relatively consistent conversion rates, how do I know if my data is statistically significant?

If I have 10 visits to each cart variation and 1 transaction, this is certainly not enough data to make a statistically significant conclusion.

Cart 1 - 10k visits, 1000 conversions, \$5000 revenue
Cart 2 - 10k visits, 1000 conversions, \$6000 revenue
Cart 3 - 10k visits, 1000 conversions, \$7,000 revenue

Surely this must be enough data to choose cart variation 3, but how is that calculation made?

How about the following extreme scenario where I'm testing the price of a poster in the cart but have very little traffic.

Cart 1 - 20 visits, 10 conversions, \$100 revenue (\$10 product)
Cart 2 - 20 visits, 8 conversions, \$800k revenue (\$100k product)

A split test calculator tells me I need 200+ visitors to reach 90% statistical significance for determining which cart converts better. Intuition says to keep selling the magazine at \$100k, as there were actually 8 people willing to buy the poster for \$100k - there are probably more. Can you make a judgement call to permanently sell it at \$100k even though the data indicates the test is inconclusive?

Any advice or pointers to a book or site would be very appreciated. I'm wishing I had paid more attention in my statistics class...

Thanks,

-Eugene

parke02

8:12 pm on Mar 2, 2011 (gmt 0)

New User

joined:Feb 3, 2011
posts:2

bump

RhinoFish

2:27 pm on Mar 3, 2011 (gmt 0)

Senior Member

joined:June 20, 2006
posts:1896

> Cart 1 - 20 visits, 10 conversions, \$100 revenue (\$10 product)
> Cart 2 - 20 visits, 8 conversions, \$800k revenue (\$100k product)

did you mean to have the "k" in the second one, and not in the first...

you have a 60% conversion rate of a \$100,000 product?

p5gal5

10:37 pm on Mar 3, 2011 (gmt 0)

Junior Member

joined:Mar 21, 2006
posts: 102