Robert_Charlton - 3:09 am on Jul 22, 2010 (gmt 0)
curioustoddler - Your post appeared just as I was in the process of answering jdancing's question (my next post), which is addressing the "follow" vs "nofollow" issue rather than the "noindex" question.
For the moment, let me say that your results are very intriguing. I'd have to know more about the structure of your site before I could say much more. I'm curious how you were handling the dupe issue before now. From your first post on this thread, I assume that you were not using "nofollow", and that we're both in agreement on that issue.
I've been recommending the "noindex,follow" robots meta tag over the rel="nofollow" link attribute for some time. Offhand, I would still recommend trying a structural fix before I used "noindex". In terms of spidering resources, I'm assuming that Google is using roughly the same crawl effort to process a "noindex" meta tag as it is to spider the page, so I'm not understanding how "noindex" is making better use of your "crawl budget", if that's what's accounting for the difference you're seeing.
The core issue initially in this thread was "follow" vs "nofollow"... but you're adding a very interesting extra ingredient.