Simple A/B testing for Google Analytics

[abtst]

Hi,

I'm working on an A/B testing solution that will report directly
to Google Analytics and you take your conversion goals right
from Analytics too.

It is easier to use than Google Content Experiments because
you won't have to create different URLs for your experiments,
there are no redirects. And you can test across all pages of
your site (or any single page too) unlike the Content Experiments
solution. Integration with your site is also easy - you place a single
tag to your site and that's it.

What do you think of it and are you going to use a solution like this?
Pricing will be very competitive too - 2-3 times less than a market
leader.

[edited by: brotherhood_of_LAN at 2:53 pm (utc) on Oct 17, 2014]
[edit reason] removed double post and personal link [/edit]

[rish3]

Looking at the universal analytics "custom dimension" set up, it seems like it would be fairly easy to do this sort of thing without a third party tool.

<!-- 3 buttons, all hidden -->
<button id='grpa' style='display:none;'>Buy Now!</button>
<button id='grpb' style='display:none;'>Buy</button>
<button id='grpc' style='display:none;'>Add to Cart</button>

<script>
// using three groups, but you can use as few or as many as you like
var groups = ['grpa', 'grpb', 'grpc'];
var randindex= Math.floor(Math.random() * groups.length);
var selectedgroup = groups[randindex];
// unhide the button for this group
document.getElementById(selectedgroup).style.display = 'block';
// set the split-test-group dimension in GA
ga('set','split-test-group',selectedgroup);
</script>

[abtst]

Hi Rish3,

Thank you for commenting.

Yes, your solution works, but:

*) Will you get the statistical validity of your experiment?
*) You will have to manually stop your experiment - dig into the code again, etc. It can't be stopped automatically.
*) You can do some Javascript development, and many people can't. Many would prefer a visual way of doing changes.
*) Plus you have to be careful here. Your code assumes that the "ga" function exists. And it depends on the order of javascript tags inclusion and sync/async firing. For quite many people it is not easy too.

*) And you have to set up those custom dimensions anyway - so you end up doing as much work, your gain

[rish3]

Will you get the statistical validity of your experiment?

What? Are you commenting on the statistical validity of Math.random() or ?

You will have to manually stop your
experiment

Assuming I don't have a content management process, I guess.

Your code assumes that the "ga" function exists.

That's silly. It's a quickly written illustrative snippet.

And you have to set up those custom dimensions anyway

Uh, yeah.
a) Navigate to Admin->Property->Custom Definitions->Custom Dimensions
b) Click "New Custom Dimension"
c) Type "split-test-group"
d) Click "Create"

[abtst]

I mean comparing performance of your 3 options, not random() of course.

By automatic stop I mean if one of the options is substantially better than the rest, then by routing traffic to the rest makes you lose your conversion. To avoid this situation you have to regularly monitor your experiment and you can't "forget" about it.
However with our solution you can concentrate on other things and the best option will be automatically selected when the system knows for sure (i.e. with a 95% significance rate or more) that there is a winner.

Doing random() is certainly not the best of the options as your loss is not optimal.

For the last item, besides d) you also have to do your Javascript coding anyway.