I wouldn't worry about a couple of raindrops in the pool that is your backlink profile. I'd focus more on improving your site and interacting with those that are interested in your topic. Guest posts are good for that, with or without backlinks.
From experience, if you focus on managing your link profile too heavily you are virtually guaranteed to make it look unnatural.
I would guess it wouldn't be a "guaranteed slap", especially if you're moving away from "keyword rich" to "example.com" or something similar.
Google recommends making an effort to have as many links as possible removed if you're hit with an unnatural link penalty and IIRC they have also recommended having links updated to the new domain when a site is moved, so I don't think a change in and of itself necessarily indicates "control" of the links in a harmful way.