I work with smaller sites with less than 100 pages and this gives a particular perspective
I can confirm that Internal semantic structure is a major issue but I think that it is more specifically focused anchor text to page linking that is doing the trick. In other words if your site is semantically on topic and has many pages each with a link with the target anchor text pointing to a page that you want to rank for the term in the anchor text then it will rank better than a page on a site where that page has more external backlinks with the same anchor text if it is not targeted with as many internal links.
Anchor text repeated on many pages within a site pointing to a page on that specific subject is working very strongly and is outweighing external backlinks.
This may be a coding error as it is just too easy to spam. The webmaster is in total control of the semantic structure of their own site. Currently bigger sites with well structured linking are beating smaller sites that are more focused with the same well structured linking. Highly targeted micro sites with many keyword anchor backlinks are not doing so well.
I wonder if Matt Cutts has finally given up on trying to persuade people from buying backlinks and is just making them worth much less for a while.