I see something similar to diberry, but I think my phenomenon that I am seeing is that people still search for those terms, but Google does not rank the site at all for those terms.
Google instead route those traffic to other websites probably based on testing and such. Because the site is not being served at all, there is no data nor "impression" ranking estimates. How can you give a ranking position value to "infinite 0"? The average between rank 1 and rank infinite = no ranking at all. So the "infinite" and no shows simply gets taken out of the equation.
"When" the site gets served to those visitors the site sits at number 1 or 2 in SERP. This greatly skews the results or report that we see. If this is what you are seeing, your report should still say that you are ranking top 5 for your keyword terms. (At least I do) At the same time those keyword traffic heads lower and lower for specific terms. Assuming that search volume stays consistent, it only means that Google is now sending traffic elsewhere.
What I experience is win some lose some while certain key phrases consolidate and breakthrough in terms of traffic volume. While the site maintains an overall "glass ceiling" or "glass bottom" which is almost nearly impossible to break.
My guess is that Google is actively comparing the user metrics between the "results" and put all sites into a battle arena. Sites that deliver "comparative" worse user metrics will lose traffic gradually. While the winner grows and eventually dominate the SERP against the loser sites.
This is based on my own observation that I search for terms that I supposedly rank number 1 on webmaster tool. But cannot see it at all when I actually search for it on Google myself. Like far into the SERP like 100's. The keyterm however is bringing a decent amount of CTR. And vice versa rank number 10 on webmaster tool but actually see it as number 1 possibly due to personalization.