Since Google weaned us off of their amazing multivariate testing tool earlier this year, Iím trying to develop our own in house A/B testing program. In hindsight, I should have done this long ago as a customized solution would have saved tons of time over copying and pasting that code for every test.
Iíve got just about everything we used with Google done now but thereís one statistic they displayed that Iím having trouble figuring out how to calculate, % Chance to Beat Original.
Given the conversion rate and margin of error (using a 95% confidence level) of two variations, how would a person calculate that?
Iím assuming itís not quite as straightforward as simply calculating the difference in means because what youíre really doing is calculating the difference between two normal distrubtions.