Are we missing something when we A/B?

What happens to the B customers?

[pavlovapete]

One thing that has struck me about A/B testing is that when we choose the A treatment we are throwing away things that the B customers prefer.

Obviously a clear winning A (whatever that percentage is) appeals to a majority of the prospects exposed to the ad.

However are we also throwing away a treatment that the B leaning prospects would respond to?

How do we ensure we cater for both A and B prospects?

[eWhisper]

It sounds like you don't want a full segmentation page, but just a way for B customer's to find what they're looking for as well.

Can you add a small box on the page that talks to customer B? So if B starts looking around, they can see the offer that is more specific to their needs?

Personally, I'd play with some designs to see if you can add some way for B to find the info that doesn't hurt the conversion rate of A visitors.

[pavlovapete]

>> you don't want a full segmentation page

exactly eWhisper - how do I tell when I am in this situation?

If I do an A/B and decide that the A treatment haven't I just thrown out a boatload of customers who prefer the B treatment?

I am assuming there is some scientific/ statistical basis for deciding to segment.

Thanks

[imstillatwork]

This is an incredibly good point...

Unless you can track individual user behavior, how can you ensure that your not ditching 1000 'B' preferences, just because 2000 visitors liked 'A' better.. I would rather have 3000 happy visitors :)

[pavlovapete]

Thanks for your comments imstillatwork.

I've obviously done a poor job of explaining myself.

Let's say we have an example campaign and we are A/B testing a landing page (could be testing the ad, offer, etc)

Of the 100 clicks, 20 convert on the A treatment and 12 convert on the B treatment. So we decide A is the winner and throw away the B treatment. Haven't we lost future customers who prefer a B treatment?

The example is plainly an over-exaggeration (32% conversion)

Since we are likely to be dealing with much smaller numbers in the real world how do we statistically decide to keep the B treatment for prospects who prefer B?

2 more realistic examples
Scenario 1: - 100 clicks, 4 A conversions, 2 B conversions
Scenario 2: - 100 clicks, 6 A conversions, 1 B conversions

Is there a statistical method to determine that Scenario 1 deserves a segmenting strategy while Scenario 2 does not?

[eWhisper]

Ok - think I understand, let me take another stab at this.

Of the 100 clicks, 20 convert on the A treatment and 12 convert on the B treatment. So we decide A is the winner and throw away the B treatment. Haven't we lost future customers who prefer a B treatment?

In this case you are testing two different treatments, and the B people do not know what the A people saw. The A people do not know what the B people saw. Therefore, there could be both A & B people who like the other version better.

In the above scenario, your goal would be that if you sent all traffic to scenario A (so all 200 clicks) you would see 40 conversions.

If you sent all traffic to B, then you would only see 24 conversions.

The consumer didn't show a preference for A over B as they didn't see both scenarios. You're only examining details in aggregate.

[Yoshimi]

You're forgetting a third group, C customers, who saw A and didn't like it so didn't buy, but who wouldn't have bought even if they had seen B. Take it to it's natural conclusion, and you will never be happy until you have a 100% conversion rate.

Businesses should never try to please everyone, they will always fail. Split testing allows us to test what will maximise our revenue, that's all, there will always be customers who go elsewhere, stressing over "could have beens" will get you no where.

[agito]

Is there a statistical method to determine that Scenario 1 deserves a segmenting strategy while Scenario 2 does not?

I think that is the key question you are asking. I don't know the answer but here are a few thoughts:

1) 100 clicks and 6 conversions is nowhere near statistically relevant to begin with (but you probably know that...)

2) If your gut tells you that a segmentation strategy might work because the versions are too similar, AND you have the traffic for it, why don't you try a multivariate experiment with a segmentation element swapped in? (or an A/B/C test)

3) If it's an adwords landing page, why not start with a user-self selection page as a landing page... EG: if you want A click here, if you want B click there... then you can have the best of both worlds. You'll give visitors the page they want; the one they're most likely to convert on, and you'll also learn something from the ones that didn't convert (which page they thought they wanted)

[pavlovapete]

thanks agito,

yep - that's the key question.

user-self selection page

That is an interesting idea - there's a PPC management firm around that does this and I've been following their experiments for a while. (there are probably others that I'm not aware of)

I think self-selection is probably the answer to my question. The trick will be to craft the ad so that it catches A and B preferences and then watch how the prospect moves through the landing page. Then craft new ads. Rinse and repeat.

Being new to Adwords it is gradually dawning on me that there are no hard and fast rules - it is more of a process of continuous experimentation and learning.

I wish I had paid more attention in statistics class.

Thanks and cheers

[EvilSaint]

Pavlovapete testing different landing pages with the different ads when doing A/B testing is a good way to be able to cater for the 'B' preference.

If you show different landing pages to people with different needs, chances of you producing better conversion results are higher.

But as Yoshimi said, it is important to keep in mind that it is never possible to please everyone.

[Meenakshi Mosaic]

I believe that testing different ads/pages for A & B would be wonderful idea to implement. Let us also know the results of this test :)