Statistically Significant?

Analyzing adwords

[ffctas]

HI,
I have been advertising with adwords since day 1. I have been generally satisified with results as they have led to a lot of sales. Heretofore, I have only been able to track with ad groups, each containing a number of key words. Now with the google tracking tool I am able to see the conversion of each key word.
I have over 500 key words and a 1% buy is very profitable.
At what level of clicks do the results become satistically relevant ( the last statistics class I took was in grad school over 25 years ago:)).
I am especially concerned about taking off high volume words as it would only take a couple of sales to bring them back into the black.
Any thoughts would be appreciated.
tom

[SlyOldDog]

Well that's a very broad question. And even if a statistician answered, it would be in language the rest of us wouldn't understand. I think the key is

- how regular are your clicks?
- do they vary from day to day? (is that seasonal or random)?
- do you get a regular conversion to sales from your clicks?
- is there any consistency to the revenues?

If you are in a thin market you will get really inconsistent results, but if you persevere you will win.

[ninhld]

if you're converting at 1%, then you'll have around 30 sales for 3000 clicks. In your case, this 3000 clicks would be satistically relevant.
I believe in the theory saying a set needs at least 30 elements to be satistically relevant.

[stevenha]

A simple 2x2 table, using a chi-squared test, could tell you the statistical significance of a comparison of clickthrough rates. For example, if you get 30 clicks from 3000 impressions, enter 30 and 2970 into 2 cells of the 2x2 table. Compare that to a different ad that, say, gets 35 clicks from 2000 impressions, so enter 35 and 1965 into the other 2 cells in the 2x2 table.

Search for an online chi squared calculator, Calculate the result, and if the p value result is under 0.05, then you've got a statistically significant difference in those proportions.

[Robsp]

Nice thread,

I have a related question. I understand the statistical overall significance of say 3000 clicks and 30 buys. The question is how accurate can you predict which individual keywords produced these results if you have a total of 500 keywords and 30 conversions on those keywords?

Say that these 30 buys were generated by 25 individual keywords (and thus many keywords have 1 conversion attached) how significant is this conversion rate hence how certain can you be that you need to invest in these specific keyword?

Any statistical wizards?

[Neil_S]

Hey Tom, how are you keeping 500 keywords active? Google's new system isn't interested in how long you've been successfully running your campaign, or how much money you've spent with them.

Any of those keywords drops below what is 'currently' acceptable for the last 1000 impressions, or if your key phrase doesn't get near 1000 impressions, then your successful Campaign is GONE!

If your only worry is how to calculate statistics, then very well done. Can you give us all a tutorial?

Neil

[SlyOldDog]

I'd be interested in knowing when a competitor was clicking on me.

A statistics test would show that up.

[eWhisper]

Ropsp,

For tracking so many keywords, you really need a very large sample to begin with. We often find that some keywords do well one month, and yet a completely different set do well the next month, so most of the keywords over a period of time are significent, but if you make too many changes each month, you might be missing out on some conversions over time.

For my most active client, I track around 1000 KWs. I generally wait to determine their significance (although I do look through this info and make small adjustments every couple weeks) until either:
1. The keyword clicks should generate x sales determined by the conversion % you are attempting to reach. (So, if you want to track 10 sales, and 1% conversion, then 1000 clicks)
2. The kw click cost is equal to double the average profit of a sale (I do occasionally lose money by waiting this long, but it's also saved me from killing underperforming keywords that suddenly have a surge).
3. Every 500 clicks for highly targeted, high cost KWs. Every 1000-5000 clicks for more general targeted, cheaper KWs.
4. The beginning of each month, I run the above reports for the previous month for 1-3. And also run a report (since branding is imporntant to us) of the entire campaign (based on ad base (i.e. google, ov, verizon) compared to sales, as a keyword that might only cost us a couple hundred a month, which loses money, we sometimes continue to run it anyway.

Note: For some campaigns, with small KW base and highly targeted words, we do combine the entire campaign together to determine signifigence, some of these campaigns only generate 100-500 clicks for a month across all ad groups.

Very imporntant, if you have a site with repeat customers. You want to determine what % of people buy from your site multiple times in a year. Then determine the profit per year of your average repeat customer. This savings (not having to pay someone to send them to your site, as they return directly) can be applied to increasing your ad dollars, taking a small loss or breaking even on your ads, as over the year, they will have spent more than what your KW cost implies.

With so many keywords, and the fact that some of the keywords I'm only looking for a .1% conversion rate (high profit/sale), it might take a couple months to hit all those conditions for any individual keyword. So often patience helps in getting your initial sample to begin tracking changes in search patterns and KW significance.

The post length got away from me, so I hope this helps.

[Robsp]

eWhisper,

Thanx for your elaborate and very useful post. As we just started using the Google conversion tracker (and did not use anything to do this before) and our product has a trial period of 14 days we have not seen enough conversions since the Google tool came out to be significant. I'll try some of your metrics to evaluate the performance of my campaign.

I have to say that this tool does make optimizing campaigns much more fun and profitable :-)