If the excessive 404s are coming from internal links, then at some point that causes trouble. Google definitely did confirm that. And it makes sense - they'd be sending visitors into a poor experience.
But they also confirmed what I'm saying about external sources of 404s - I believe it was John Mueller on their Webmaster Forum. How could you be responsible for that? And if you try to get a "perfect report card" in WMT, it will definitely drive you bonkers.