Can I Truly Splittest?

Ads have different CTR

[One Thing Well]

So I run a splittest.
Only 2 Ads in the same Ad Group with only 1 variable just like I'm supposed to.
The Ads have different CTRs.

I run the results to see if they are statistically significant.
Sometime they are, sometimes they are not.

So here is my question.

Do the CTRs shown reflect the actual CTR or an "adjusted" CTR based on position the Ad ran in? Wouldn't the Ad with the lower CTR start showing lower in the rankings thereby lowering its CTR even more? Does this compromise the validity of the splittest?

I think it would. Unless the CTRs are adjusted for rank, whichever Ad starts out better is going to show up higher in the listings and thereby garner a higher CTR.

Please help. This is really bothering me as I'm a splittest fiend!

[migriffin]

My belief is that it is nearly impossible to split test on Adwords since adrank (CTRxCPC) is normalized based upon position. The best way to split test, is to bid sufficiently high that adrank = 1 for all ads despite CTR.

[inasisi]

The average CTR for the keyword is used for determining the position and not the individual CTR of the Ad. So even if two ads for an AdGroup have widely differing CTRs, they are still shown at the same position. So split test is still valid.

[One Thing Well]

Thanks Inisasi,

It's amazing sometimes how all these brown things with leaves get in the way of the forest. To confirm your assertion, I went to Adwords Help which states "The Quality Score is determined by the keyword's clickthrough rate (CTR), ..."

Sometimes it just helps me to think out loud.

Perhaps this helps you out to Migriffin?

[migriffin]

That doesn't seem to be correct in my case. I have several ad groups where I was trying split testing of multiple creatives. A single keyword will trigger different ads at widely different positions. If the ads appear in different positions with the same keyword, split testing is invalidated.

[eWhisper]

What are you trying to test?

CTR
Conversion Rate
Profit per impression
Time spent on site
etc.

Without knowing the purpose of the test, it's difficult to determine if the setup is done properly.

Old Post on Testing [webmasterworld.com]

[eWhisper]

Oops forgot to answer the question - the CTRs shown aren't normalized by position.