Apart from penalizing for mentioned blackhats and inorganic SEO Google also takes manual action against sites that have performed over optimization. Basically the whole concept has moved towards being ethical and natural as well. 2-3 months back I ranked a new site on 1st position and believe me I am not working for that site since 2 months but there has been just minor shift of the rankings which is from 1-2. The only thing I do is blogging on weekly basis or occasionally. If to see then really I haven't worked much with optimizing the site offline as several known methods I haven't even tried for the site.
I strongly believe that if you will just focus on building traffic through natural ways and make sure that you justify properly the links that you will build towards a site then you easily get rankings and will not be penalized.