A/B tests sample size

[humm0s]

Hi,

Hope someone could help me out.

Before I created my A/B tests on Facebook, I used Optimizely's sample size calculator and found I needed a sample size of 5,000 per variation to reach 95% statistical significance. While I was setting up my campaigns, Facebook said the potential reach will be 50,000 so I thought I was safe.

However, my campaign, which is 2 weeks long, is about to end and I only reached <300 per variation (which is well below the sample size needed to reach stat sig).

What am I doing wrong & how could I avoid this next time? Thanks in advance for your help.

[not2easy]

Hi humm0s and Welcome to WebmasterWorld [webmasterworld.com]

I can't help much with FaceBook specific A/B testing, but in general the smaller the sample, the less information you can determine from the results you see. Like any data, the larger the source of data the more useful or actionable information it can deliver.

It looks as if your campaign term was too short which gave you too small of a sample size. I would not be surprised if generic marketing text overestimated the "potential reach" since that would vary significantly on your target demographics and on the topic interest of your campaign. It would be hard to calculate accurately.

An older discussion here on A/B testing for AdWords (before it became GoogleAds) goes into more detail: [webmasterworld.com...]

HTH