Forum Moderators: martinibuster
When I launched the site I put it into a couple of forums for reviews in order to get some feedback. This was very helpful and I got tons of feedback. I made a lot of changes some of them right away and other changes were a bit more low priority.
One the low priority changes was changing the look of my Adsense ads. I’ve had several posts saying that the ads looked bad, because the ads were not in the same colors as the rest of the site. The ads were default Google Adsense colors.
Making sure that the Google Ads are an integrated part of your site and look like the rest of the site is not revolutionary news. It is an old Google Adsense tip and you might have heard the tip before, but I think this is interesting because this is an example of the exact result that a minor change like this can do. I have nothing to hide so I’ll give you the exact numbers of the effect of this very small change.
The numbers are based on 4 days before the change and 4 days after the change. Here are the numbers:
Before:
Pageviews 882
Clicks: 1
Click through rate: 0.11%
After:
Pageviews 1125
Clicks: 8
Click through rate: 0.70%
The numbers are not huge because the site was just launched, but I think they are large enough. This was a serious shock to me. A minor change like this increased my earnings 7 times – pretty wild.
I hope you can learn something from this, I certainly did. Let me hear your comments or if you have similar stories. It would also be interesting to hear if 0.70% is high or low compared to others.
<snip>
Regards,
-Zach
[edited by: Brett_Tabke at 2:45 am (utc) on April 19, 2005]
[edit reason] No references for people to find your site please [/edit]
You probably are going to see a benefit from what you did, but your numbers are crap, frankly. You can't tell a thing from such tiny numbers.
It's like saying someone is a phenomenal baseball hitter because they have two hits in a game. Sure, they might have hit .500 that game but you're dealing with the statistics of small numbers. Even the best baseball batters routinely have zero hits a game.
I hope you can learn something from this
When you launch a site dedicated to estimate the number of visitors, you should have some basic knowledge of mathematics and statistics. If you had that knowledge, you certainly could estimate the error bandwidth for an AdSense sample with 1 click, and a sample with 8 clicks.
You mentioned that many visitors to the site were people attracted by URL dropping in forums. These are not genuine visitors. Furthermore, AdSense needs some time to select the right ads for a new site. It is therefore normal that a site will show PSAs in the first days. Maybe your low click count on the first day was caused by PSAs. So IMHO opinion it is impossible to compare the figures you mentioned and conclude that you made "major" improvement which we can learn something from.
Some background on the significance of your figures without taking into account that your visitors were not average visitors and that possibly PSAs were shown instead of regular ads: Clicking on an AdSense ad can be seen as tossing a coin. The standard deviation for this is called the binomial standard deviation and as you can see in [webmasterworld.com...] message #6, this can be estimated by the square root from the number of clicks.
Let's say that your average number of clicks per day is 4.5 and 1 and 8 are the extremes. In that case:
68% of the days you have between 2.4 and 6.6 clicks
95% of the days you have between 0.3 and 8.7 clicks
99% of the days you have between 0.0 and 10.9 clicks
Because the estimation with the square root does not take the discrete values of clicks into account, these ranges should be rounded but you get an idea. So decide for yourself if what you have seen is a significant improvement, or just caused by random effects.
You probably are going to see a benefit from what you did, but your numbers are crap, frankly. You can't tell a thing from such tiny numbers.
You would be right about this if he only was observing for a week or more. You sure can determine a trend even on low amounts of data, if the data is consistent for a given amount of time. If 1 click was the average click for the whole month or so, his statements are valid for his particular site. If he saw 8 clicks consitently the next month, then it simply means he really did get 8 times more clicks. Statistics is based on averages after all, and it is all relative.
[edited by: chopin2256 at 11:04 pm (utc) on April 18, 2005]
You would be right about this if he only was observing for a week or less
The observations were over two periods of 4 days each, and if I read the post correctly the numbers are the totals of these days, not the averages, but maybe ZachH can clarify this.
I think there is a significant difference in the two numbers. The click thru ratio went up over 600%. This is based on nearly a thousand impressions in each instance.
The problem is that there has been no correlation between the change and the increase. A better experiment would be to have both page formats shown to users on the same days. Even alternatinge the ad styles between page views. See which ad styles resulted in clicks. That takes out the time dependence of the study. For example, people may have clicked the ads because they looked different than ordinarily viewed, not because the ad style was more clickable.
No offense but the whole basis for not using math there is because statisticians would laugh at the idea that anybody would do the math. You can't tell anything based on 1 - count them, 1 - click.
If someone had sneezed and clicked a second time, you no longer would talk about a 600% increase in click-through, but one of half that. And the fact that a single tiny insignificant action like one click can alter the numbers so drastically, shows you that there is no basis for meaningful statistical comparison.
Find out that all 8 clicks were the same person and you would look downright foolish in making any comparisons, in fact.
C'mon guys, let Zach be happy with his results. If his conclusions are premature, I believe he'll know that given enough time. When I started with AdSense a few weeks back, I was like watching my stats by the hour and tried to rationalize from there with every change I did. I got over it after a few days. Zach's comments are probably more on the side of "excitement" rather than pure thought.
How large a sample should one have before being able to conclude anything?
Should one look at the number of clicks or should one look at the number of impressions?
Thanks,
For instance, you might think that because my earnings last week were 100% better and clicks 60% better than the week before, I might have done something to improve things. Nope, nada.
And that's with 120 times the traffic sample you're using here...
Now, if I *had* changed something I'd be attributing the results to the change, when it reality the change did nothing. Similarly, if the weeks had been reversed - huge drops - I'd be scrambling to change back what I'd done, when again it would have just been a normal day in the AdSense rollercoaster. That's one reason I've given up trying to tweak my ad placement or colors... I can never tell if what I've done has helped, hindered or done squat.
Then you analyze the statistics.
The math can get a little more complicated but the as a first-order approximation, you then find your margin of error by taking 1 over the square root of the number of clicks.
100 clicks and your margin of error is 1/sqrt(100) or 1/10, or 10%. If you saw 103 clicks with the other test, well, you can't tell a darn thing about the test. If you saw 125 with the other, well, that's beginning to look like a better option.
However, you must be very very careful in interpreting the numbers because it only points to statistical probabilities, not certainties. 100 clicks on your A test vs. 125 on your B test might have been a statistical fluke. If you were to repeat it so that you had millions of clicks, you might find that B was in fact a worse option than A but that it only looked reasonable with a more limited and error-prone sample.
