Welcome to WebmasterWorld Guest from 50.16.112.199

Forum Moderators: buckworks & eWhisper & skibum

Statistical Significance

   
10:53 am on May 19, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



How many clicks should a keyword receive before it can be considered statistically significant for purposes of ROI tracking?
11:47 am on May 19, 2008 (gmt 0)



There's no simple answer to this question, other than: it depends on how reliable you need the results to be.

If you have a target conversion rate of 1% then you would likely need in the region of 150-200 clicks without conversion before you could consider disabling the words with an acceptable degree of reliability - depending of course, on what is acceptable to you or not.

The actual total number of clicks is not that relevant - how many you would look to get depends on what your goals are, and how confident you want to be that you have enough data to make decisions based on it. Clearly, if you want a 50% conversion rate you will know much quicker whether those words work for you or not.

Sorry if that sounds a bit vague!

11:55 am on May 19, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



OK, that makes sense. How about a 2% conversion rate? Maybe 100-150 with an acceptable degree of reliability?
12:00 pm on May 19, 2008 (gmt 0)

5+ Year Member



[URL deleted]

Roughly speaking, it depends how you model it and then the confidence level you're willing to accept (given the CPC you are paying on the keyword in question).

I would agree with the above post and say that for me (avg. CPC ~0.20 and target conversion ~ 2.5%) after 100-150 clicks i would get rid of the keyword.

[edited by: buckworks at 12:23 pm (utc) on May 19, 2008]
[edit reason] URL deleted [/edit]

12:07 pm on May 19, 2008 (gmt 0)



Again, how reliable (what degree of confidence) does it need to be? If you're paying 50 a keyword, the you're likely to err on the side of caution. If your keywords are at the 5p level then you would likely go for a much higher degree of confidence.

I can't give you an easy calculation without it being misleading, because of all the variables involved.

I'd imagine 100-150 clicks with a desired 2% conversion rate would be more than reliable enough for most purposes, if the listings received no conversions at all. I'd say perhaps too reliable, especially if click cost is high.

2:50 pm on May 19, 2008 (gmt 0)

5+ Year Member



You need to conduct F tests.
3:17 pm on May 20, 2008 (gmt 0)

10+ Year Member



I've battled with this question for a long time and basically came to a non-scientific conclusion, that it has less to do with click volume than I originally thought.

Say you have two business owners. They both pay for the same keyword that gets 100 clicks over a one week period at a $1 CPC. They each pay $100.

Business owner A, gets 2 conversions at $300 aov yeilding $600 for a 5x ROMS (600 - 100)/100.

Business owner B gets 1 conversion at an aov of $150 for 50% ROMS (150 - 100)/100.

If the question is, how much data is statistically significant enough to take some sort of action, than Business owner A is most likely going to say "Raise my bid."(to what is a different question) While Business owner B will most likely, depending on his margins, say "Eh, lets wait and see, i don't feel there is enough data to say this keyword will, on average over time, perform at this level."

For keywords that don't get any conversions, I've historically gone with the rule that, if your site converts at 1%, then you need at least 100 clicks, and probably 300 to account for other friction/seasonality and unpredictable randomness that will never be known. But again, if the margin on just one widget is < $100, at a $1 CPC, you've already lost. But if your margin is $300, then keep running.

 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month