and google does do this right? Drop your site for not building fresh links even though its doing well?
Google doesn't expect you to actively "build" links - they expect that a good, active website will naturally show some signs in links, or mentions, or other types of citations. This expectation appears to be quite different for different types of websites, too.
Unless your site has not acquired any new links for many, many months, it's more likely the ranking changes are related to something else - even an algorithm change of some kind.