Thanks for your thoughts.
You're right, in my case it is "link pruning". More than 80% of my backlinks come from weak directories, different kinds of directories.
The directories itsself are not really spammy (except some DMOZ-clones concerning one site), but they have no rankings and traffic, most of them since any panda or penguin update. My entries are all good, i.e. unique texts and providing good content, changing anchors.
BUT: this weak backlinks are more than 80% of my backlink profile, they are all from directories (one kind of source), have all "PR unranked", missing all relevance and trust, have all KW anchors --> penguin catched me for all important KWs.
My plan is to get rid of at least 50% of my directory links, the bad ones of course, keeping the "better" ones. As there are several sites and several important subpages, building new good links alone is not a way for me. I have to remove and disavow.