Maybe the winners among the affiliate publishers are making up for what the losers have see evaporate. If the networks ARE feeling a significant negative impact, then they should be getting vocal.
I agree, it has to be worring time for these companies. I was one of the larger affiliates and there are no other affiliates in the place of my results. The money I have lost for the affiliate company is sure to have an impact on their business.
I do however question a sitewide Panda penalty
Is panda a penality though, I think Google will tell you no, although I agree any sitewide penality has to be frowned upon unless you are harming the public. The penality should be for pages they check, how can someone say there is something wrong with your whole site after a manual check of 3 or 4 pages.
If it is linking that causes a penality, is the penality caused by Google devaluing all your links, thus you are the same as having no links, so you rank accordingly -50? I have never heard that this is the case, but a close friend that had a -50 penality for a few years began ranking again after talking another webmaster to link to him with a PR5.