I'd remove noindex tags on your own pages since it won't change their evaluation one way or another. Google knows about the pages even if you don't want them indexed and they are more concerned with what they are sending a visitor to in general. A site with loads of blocked pages (noindex or robots.txt) is likely not as trust worthy.
Pages on your site with noindex still have ranking value, pr etc. They aren't excluded from the evaluation process, they are simply dropped from serps.
A falling number of backlinks is a condition affecting most webmasters I've spoken to recently, sites that used to link to them still do but are devalued. This isn't a horrible thing unless a lot of your links were from places that have been devalued, in which case it can be devastating.