When I say fresh backlinks, I'm creating links gradually from relevant sources as everybody else is doing.
That answer is a bit too casual for me. "Everybody else" is doing a bajillion different things, and the most common reason for a penalty is something in the backlink profile that Google see as an attempt at manipulation. It even looks like too much use of the same keyword in the anchor text can trigger a penalty.
Another common reason for a drop in rankings is a hacked website that now hosts parasite links. Some hackers are also quite devious these days about cloaking the parasite content, so I'd encourage you to make a through investigation here, and use "Fetch as googlebot", not just your own browser.