Well... many people believe that the old adage that there's nothing a competitor can do to harm your site is simply not true. But yeah, I think if having _inbound_ links from bad sites/neighborhoods actually caused a penalty as such, that would make for a really easy way to nuke your competition.
Furthermore, most people who rank for any common keywords are probably getting large numbers of inbound links from zombie-hosted scraper sites. I know I have lots of them, as do all of the top sites in my niche. If that caused a penalty, we'd all be hosed.