Forum Moderators: open
I need to find the sample size, but I am not sure if it can be
considered a random sample since we will not be selecting the subjects
at random, they will be selecting us by choosing to click on the ad.
Further, since market conditions change over time, a sample that
represents the population now may not represent the population a few
weeks later so the test may be useless for projections. So, am I
testing a sample or am I only testing the entire population for a
shorter period of time?
I am not a statistics expert so please let me know if my analysis is
correct...
An Overture ad for a given keyword costs $10.65 per click for the top
position. I hypothesize that we will maximize profits by paying for it
and letting it run continuously.
We can test this theory if we can commit to paying for enough clicks to
equal the sample size of the total possible clicks for a given
click-through rate (CTR), for a given time period. Overture provides
estimates on how many impressions we will have for the month so once we
determine the CTR for our ad, we can estimate the total number of
clicks we would have if we let the ad run for an entire month.
The total number of clicks for the month is the population size (notice
that the population size is not the number of people whom see the ad,
but it's the number of people whom click on the ad because we are
testing the performance of the Web page).
For example, let's assume a 2% click-through rate and 612,740 ad
impressions for the month so at a 2% CTR, that would result in 12,254.8
clicks for the month. If you set the population size to 12,254.8 with a
confidence level of 95% and a confidence interval of 2, the sample size
would be 2,008 clicks at a total cost of $21,385.20.
If the conversion rate for the sample is 4.93%, 2,008 clicks will
result in 98.9944 conversions. Because we are using a confidence
interval of 2, we can be 95% sure that if market conditions do not
change and we let the ads run for the entire month, for the entire
relevant population, we would have a conversion rate between 2.93%
(4.93%-2) and 6.93% (4.93%+2).
Please poke holes in my analysis and let me know if I making any false
assumptions, any leaps of logic, etc? Or, please let me know how to
change the test to make it more valid.
The basic theory holds water, but there are a couple of tangible variables that can skew your data, and intangible variables that can also skew your forecast.
Tangible variables - CTR will change based on position change for a keyword in Overture. Your position can change several times in a single day. Using the averaged CTR works fine for short term projections, but over time, competitors can jump into/out of your position/bid area, skewing your average CTR.
Intangible - Day of week has been found to signifigantly impact CTR for various vertical markets. Education (2/4 year tech schools, UOP, etc) for example has been found to have higher CTR and lead generation during week days, but terrible CTR/lead generation on the weekends.
I'm not sure why you are so focused on trying to map a forecast on Impressions/CTR -> conversions.
If you focus on CPC (Cost per Conversion) or ROAS (Return on Adspend), everything else will fall into place.
In other words, it doesn't matter if you achieve a 3% conversion rate, if the Cost Per Conversion is so high, that the keyword is not profitable.
We focus on Cost Per Conversion, setting thresholds for evaluation based on either a certain number of clicks, and/or incurring a specific cost for a keyword. At that point the keyword is evaluated against the Target Cost Per Conversion. If the keyword over performs against the target, we increase the position, if it underperforms, we decrease the position.
We have found through a few years of historical tracking, that the same keyword will perform in a specific position one week, and the next week it won't. The key is to have a system in place that can evaluate the keywords at the proper time, and make the proper bid decision. This isn't difficult to do manually across a small subset of keywords, but requires an automated process if you are managing hundreds/thousands of keywords across multiple paid search campaigns.
