You don't say exactly how long you've been online with this particular site, so my observations may not directly apply. But here's what I've seen with Google a number of times over the past few years:
- A new site goes up that is well designed -- meaning validated code, good text content, not much duplication, some inbound links, no black-hat SEO.
- After a reasonable time the Googlebot finds the site, indexes it, then shortly thereafter it starts showing up in the SERPS with surprizingly strong position.
- The site owner starts to see good visitor stats and becomes very hopeful. It is at this point that I say "don't quit your day job".
- Then at some further point down the road, the site drops back to a lower level, and panic sets in, with everyone wondering how that can be, as any additional changes were (seemingly) minimal.
So I've come to the conclusion -- rightly or wrongly -- that as a matter of policy Google will often give new, well designed sites a boost at the front end, but then after a difficult-to-determine time period the algo will re-position them into another slot in the "natural order" of things.
As the months/years pass, if that site continues to attract quality inbound links and continues to add good quality content -- so they become more of an authority -- then they will rise again. If not, then they tend to stay back in that lower position.
So what throws people off is the "Google Gift" at the start ... but to paraphrase the old saying, "what the [Google] gods giveth, they can taketh away".
So as I said, this may not apply to you if you've been online for a few years, but on the other hand if your site is relatively new, you may be falling into the pattern.