Google AdWords

This is the setup for a very simple test to start people thinking about how to split test and measure such test results. There are, of course, much more complicated techniques to use. Most webmasters/marketers don't have incredibly sophisticated tools, so this example is based on a low tech variation that can be run with just an excel spreadsheet.

If you think about an ad, you have 4 lines to work with. (Some consider it 3, depends on how you feel about the Display URL. Each line is both it's own entity, and related to the whole ad - all at the same time. Each line must carry it's own weight, and reinforce the message you're trying to send via your ad.

Note: This test is for using 3 lines. 1 title, description line 1 & 2. If you like writing sentences (often useful if you're in the premium position), then you'll be testing the ad copy slightly differently depending if the sentence can be broken into halves and split tested using the below.

This is assuming we're only testing ads, and not landing pages. It's also assuming there is not a tag/branding line we want each ad to have (although, if you're still trying to figure out how to brand your site and looking at tag lines for every ad - this is a technique that can help you find a line that 'speaks' to people in your market). I'm going to ignore the display URL right now, as you should get the idea from the rest.

Turn off 'ad optimization' in the campaign settings, we want all the ads to receive equal exposure.

I would suggest ignoring dynamic insertion on the first trial run. This can be added later after you determine what type of copy works best for you. But for dynamic insertion to work properly, you also need to know what the 'backup' text needs to be as it will be shown sometimes.

Determine how many ads you can successfully test based on your volume (if you're getting 10k impressions a day, you can test many more ads over the course of a 2 week trial than if you only get 100 impressions). At some point, you're going to want to limit the total number of ads you're running purely from an unwieldy amount of data standpoint.

Create 2-4 titles (T below), description line 1s (D1 below), description line 2s (D2 below), and then mix and match them. Remember, these go up exponentially, testing 3 lines is 27 ads (i.e. #t x #d1 x #d2).

Your ads will look something like this:
ad# T D1 D2
1. 1 1 1
2. 1 1 2
3. 1 2 1
4. 1 2 2
5. 2 1 1
6. 2 1 2
7. 2 2 1
8. 2 2 2

Using just 2 variations of each line, we've already created 8 ads, you can tell this number goes up quite quickly, a test of 4 different copies can quickly become a huge amount of work.

After you've hit your time frame for testing, put all your data into an excel spreadsheet. (I prefer a time frame of at least one week so I can receive visitors on every day of the week to test out the results. Often, I go for 2-3 weeks. If there is a holiday or special event in your week, then you should go another week as holidays lead to a different type of traffic. Some prefer to test by total number of impressions - it's all preference).

If you're using the Google conversion tracker, select 'custom report'. Deselect 'keywords' as you don't want to see every keyword. Select the AdGroup you want to test (you can also select several if you want to test how different lines work across your entire account - useful for testing 'tag lines') Select CTR, conversions, conversion rate, transactions, transaction rate, ad title, ad description line 1, ad description line 2.

Run reports by CTR, conversion %, total conversions, total transactions, transaction rate and by total profits (one ad could have a lower CTR but higher conversion %, thus produce a better bottom line (Forget ROI, Just give me profits). To get the number for total profits, you'll probably have to use information from your intranet/shopping cart or use the advanced conversion tracking options and use your purchase variable in the tracking code.

Transaction rate is a useful number to use. It shows how many people were either repeat customers, or came back after visiting your page and then converted (conversions are for people who visited and purchased without leaving your site, transactions are the total number of times your page with the Google tracker & a cookie on their machine saw the "thank you" page). If you see ads with much higher transactions than conversions, you might want to find ways of converting them more quickly before they leave your site.

You're going to download this report from Google using the CSV for excel link.

First, run these reports by all ads. This will give you a benchmark to look at for the average conversion % / bottom line of the aggregate numbers.

Then you'll run them by title, description line 1, description line 2. In the above, you'll have 12 reports (title 1, title 2, description line 1A, description line 1B, d2A, d2B, ads 1-6). Analyze these numbers carefully. Don't just see which lines received the best ROI, profits, and CTR - look to see if one description seemed to work better with another description line or title line, etc.

In excel, a starting point to run such reports is to use the autosort feature. From the dropdown box choose to only show the line you want to measure. The autosum feature will then let you see exactly what that line received. For easy of use, copy that line into a different sheet along with what title/description line you were measuring(remember to paste variables only, not formulas). This will store all this data in one place for ease of analysis.

At this point in time, you should know exactly what lines gave you the best conversions, best CTR, highest bottom profit line, which are the second best, etc.

Optional step: Take the best performing ads (highest CTR, highest ROI, highest profits, highest conversion % - odds are, you won't have 4 different ads, one ad will be the best for more than one), and if applicable, apply dynamic insertion to the title (for some ads, you'll also be able to test dynamic insertion on lines 2 and lines 3).

Repeat the above as often as you desire. Of course, you're going to always take the best performing ad or 3 and keep using them in the split test for benchmarking purposes.

If you want to split test landing pages, this is a good time to start.

First, remember:
1. Ads and landing pages have a relationship to each other.
2. Different ad & page combinations can lead to different conversion %s.
3. It's possible your best ad and best page actually have a lower conversion %, but your second best ad and best page (or vice versa) produce better results.
4. Don't assume you know what is best or what should work, let the numbers tell you the story.

If you want to test pages at the same time as ads, then once you make your ads, duplicate them exactly except for the landing page. (i.e. In the above example, you'll have 8 ads to one landing page, and 8 identical ads leading to a different landing page) for a total of 16 ads testing two landing pages. Again, these are exponential numbers. If you're dealing with low volume, choosing your top 3 ads and testing them with 2 landing pages might be better as you only need enough information for 6 total combinations (3 ads to landing page 1, 3 ads to landing page 2) and not 16.

When you begin to analyze the results, you are going to add another step in the reporting, landing pages. You'll look at the overall conversion % of a landing page as a benchmark, and then compare how each ad also interacted with that landing page.

The results of such split testing can be very eye opening. At some point in time in your split testing, try making extremely similar ads with just a one word difference (a plural and a singular; two different 'offer' keywords; two different adjectives; etc.) At this point in time, you're refining your ad copy and offers as much as split testing to learn more about your audience.

Split testing ads and landing pages creates additional work. This is very worthwhile time spent to get the most from your advertising dollars. Having access to toolsets can definitely speed up the process, although, it's very possible todo all the required analysis with free Google tools and just a spreadsheet.

Hope this begins sparking the thinking process behind moving from striving from just high CTR ads to finding the best converting/profit margin ads.

Related: A/B Split Testing (in supporters).

For some accounts, it's better to first get the keywords set up with 3 ads in each group to determine an initial benchmark for conversion rates, keyword CTRs, and what ad copy keeps your keywords from being 'in trial', 'in hold' etc, and to build a healty CTR for the campaign. This helps when you beging to start testing that you're not also fighting the Google system from disabling some keywords.

For others, I build in the split testing with the client to begin with (this is often for brand new sites where there are no stats, conversion data, etc) so that we can immediatly start to see the patterns associated with the account and adjust pages, ads, and keywords appropriately.

Of course, the amount of data you get to play with (impressions, clicks, etc) also determine a lot about what you can test.

Good post. We do this stuff all the time but I picked up a few new ideas from your post.

We usually test 3 to 5 ad creatives and after analyses continue with 2 and add 2 new ones along the same lines. It is amazing what small changes can do to already good copy.

One of the more interesting questions is how much clicks do you need on a given ad to have some statistical significance to your results. We usually do not make any descisions below 300 clicks. How do you handle this?

As usual, the answer is it depends. If you have a conversion rate of 1 in 2 clicks, you should be able to decide after fewer clicks to make it statistically significant. But on the other hand if the conversion rate is 1 in 200 you might need a lot of clicks to make this decision.

Reminds me of a thread from a year and a half ago:
http://www.webmasterworld.com/forum81/1341.htm

It's always interesting comparing what you did then to what you do now. Think I need to add some 0s to how many KWs I track - however, a lot of the information there is still quite relevant.

For larger spends, the above post is still quite relevant.

However, as more and more SMEs enter the PPC market place, one thing I've changed is how to measure their statistical significance as they can't afford (or are too impatient to wait for) some of the longer tests.

I still like time frames. I think that measuring at least 2 weeks gives you a better day by day picture of search activity. Of course, one has to take an intuitive approach to certain vendors (i.e. seasonal items, accounting software, cards, jewelry, etc) However, sometimes I now lower the number of clicks to 200-300 as long as they occurred over at least a week's time.

For some of these companies who are only spending $200-$500 a month, to adequately measure their success also requires 'grouping or theming' of highly similar keywords (which goes directly to AdGroup organization). This is where running analysis data of AdWords along with Overture numbers can be of great use. Because of how Overture handles KWs, there are less total variations available (misspellings, plurals, etc), and running OV & AdWords reports independently, and then aggregating the data together to see a larger volume set in KWs can be enlightening.

300 clicks! That's like trying to get to the confidence levels for survey research. PPC (as are its cousins in direct marketing) operates on a completely different statistical basis.

The old DM analyst's rule of thumb is that you have good enough statistical confidence on a data set with 7 buyers (the universe size is irrelevant). My observation is that that rule works just fine in PPC.

At an average 1% conversion this means you need about 700 clicks in your case. I took the 300 clicks indeed from my previous survey stats experfience and agree that PPC operates on a different basis. There is very little data as to on what basis that might be though (and very little discussion about it too).

Any more insights?

could you expand on this?

PPC (as are its cousins in direct marketing) operates on a completely different statistical basis.

Robsp, it sounded to me like you were trying to calculate survey stats. That's the wrong branch of statistics. You are using the wrong confidence tests. Survey analysis is in a branch of statistics where the thing you're counting represents a large portion of the sample universe. In PPC, direct marketing, and manufacturing QC the thing you're counting is a tiny portion of the sample universe. There are different statistical tests under these circumstances.