homepage Welcome to WebmasterWorld Guest from 54.237.95.6
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld
Home / Forums Index / WebmasterWorld / Ecommerce
Forum Library, Charter, Moderators: buckworks

Ecommerce Forum

    
Shopping Cart Multivariate Testing - Statistical Signficance
parke02




msg:4262010
 1:55 am on Feb 3, 2011 (gmt 0)

Hi All,

I'm generally familiar with the concepts behind A/B vs MVT testing and measuring for statistical significance when it comes to testing pages, but would somebody please explain how to take the revenue into consideration for testing a shopping cart variations?

For instance, if I'm testing shopping cart variations with various cross sells, or complementary products, and the test yielded in relatively consistent conversion rates, how do I know if my data is statistically significant?

If I have 10 visits to each cart variation and 1 transaction, this is certainly not enough data to make a statistically significant conclusion.

What about the following scenario:

Cart 1 - 10k visits, 1000 conversions, $5000 revenue
Cart 2 - 10k visits, 1000 conversions, $6000 revenue
Cart 3 - 10k visits, 1000 conversions, $7,000 revenue

Surely this must be enough data to choose cart variation 3, but how is that calculation made?

How about the following extreme scenario where I'm testing the price of a poster in the cart but have very little traffic.

Cart 1 - 20 visits, 10 conversions, $100 revenue ($10 product)
Cart 2 - 20 visits, 8 conversions, $800k revenue ($100k product)

A split test calculator tells me I need 200+ visitors to reach 90% statistical significance for determining which cart converts better. Intuition says to keep selling the magazine at $100k, as there were actually 8 people willing to buy the poster for $100k - there are probably more. Can you make a judgement call to permanently sell it at $100k even though the data indicates the test is inconclusive?

Any advice or pointers to a book or site would be very appreciated. I'm wishing I had paid more attention in my statistics class...

Thanks,

-Eugene

 

parke02




msg:4275468
 8:12 pm on Mar 2, 2011 (gmt 0)

bump

RhinoFish




msg:4275908
 2:27 pm on Mar 3, 2011 (gmt 0)

> Cart 1 - 20 visits, 10 conversions, $100 revenue ($10 product)
> Cart 2 - 20 visits, 8 conversions, $800k revenue ($100k product)

did you mean to have the "k" in the second one, and not in the first...

you have a 60% conversion rate of a $100,000 product?

p5gal5




msg:4276272
 10:37 pm on Mar 3, 2011 (gmt 0)

To acquire a statistically significant result, you will need one either:
1. A greater difference in the outcomes
2. An extremely large dataset
3. Less variance within each group (site)

90% statistical significance is not very high - usually 95 or 99 are considered benchmarks.

As variance is a determining factor in statistical significance, if your transaction values vary (you carry more than one product at one price), it will take longer (more visitors) to achieve a meaningful result. There are lots of books on this, but they are textbooks (and I'd recommend some advanced calculus before tackling them).

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / Ecommerce
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved