I understand exactly how you feel as I have these same thoughts. I think it's made worse by the fact that I don't have a team of people reviewing my sites so sometimes I question myself too much.
Netmeg has more patience than me as I try to hold out for at least 7 days when running a test. I don't have a magic number for page views you should be looking for before you decide if whatever you changed is a keeper or not but the more you have the better.
Also, try not to test too many things at once, make one or two changes to the layout then evaluate results.
What I know I have learned is that sometimes very small changes can make big differences. Something as simple as moving an ad 50px can have significant results, changing a line of text to bold, taking something you think looks cool off the page, adding something useful to the page.
When you have a busy season or high traffic period probably a good idea to give it a break and enjoy the results for a while as Netmeg suggests.
I doubt I'll ever stop testing.