Has there been a major turnover for the other top sites for those keywords? Or is it just your site pages that have dropped?
When I say fresh backlinks, I'm creating links gradually from relevant sources as everybody else is doing.
Can you define "gradual" please? How many links per month?
Most of my links are in-content links from high PR sites, high PR pages...
Maybe the PR is TOO high? Are you buying these links? If not, could it APPEAR to google that you are buying these links?
Maybe if the top ranking sites have poor links, google EXPECTS sites in that niche to have poor links, and would be suspicious if one site had a monopoly of high PR links?
Have you looked at ALL the backlinks to your site to see if a competitor has possibly placed thousands of links to your site without your knowledge?
As for penalties, I haven't experienced any myself (knock on wood), but many of the people who I have spoken with - and who have YEARS of experience with this - are convinced that manual penalties are very real.