Making sense from A/B testing

[dickbaker]

I have a couple of pages with a new design that I'm testing using Google Analytics. I'm doing A/B testing of them against old versions of the pages (widgets.html versus widgets2.html).

The problem I see with this approach is that someone who's going through a site that has one look to it may be a little confused when the look changes when he visits another page. I know I would be.

If this is true, then the page with the old design would have an inherent advantage over the new designs, making analysis difficult.

Does anyone else believe that, or does anyone have any experience with this? I'm wondering how to take this into account when evaluating the new designs.

[Receptional]

Ha!

Sometimes the most fantastic of technologies (ab testing) fall down at the first hurdle! You make an immensely valuable point!

The user experience is as much about the journey as it is about an individual page. I guess that the system is really only good at much more specific tests, like 'should the button be red or green'?

If you Want a massive change, then I guess you need to test on a landing page, exclude return visitors, and work from there?

[makemetop]

Yes, this is a very good point. I usually advise two stages of A/B testing. The first, as Receptional says, is to carry out some simple variants of the landing page without making more fundamental look and feel changes. The second is to identify the most commonly used "buying path" through the site (not easy with a large site) and create almost a series of small sub-sites for PPC A/B testing with different page layouts for each sub-site, to show consistency on the buying journey.

If you can do this, it can give invaluable information on ultimate design for the whole site, but, obviously, is not easy to do on every site. But where you can do it, well worth the time and effort.

[dickbaker]

I wonder if there's really any way to predict user behavior.

I A/B tested several popular pages to measure page view time and bounce rate. They were very close, with the new design having a slight advantage. So I converted all of my pages to the new design and uploaded them about 1 am this morning.

Today I find that my online store page, which is typically #4 or #5 for page views, is now #14. My guess is that repeat visitors are a little confused with an entirely new look, and are a bit reluctant to go shopping. I don't know for sure, though.

[Receptional]

Maybe we still need need eyetracking or user observing to see this. I am sure we would all be surprised what we'd learn from getting five people off the street and ask each to perform a presumably simple task on the site. All the A:B testing in the world can't make up for that confused or confounded expression on their faces.