I think "Penalty" is the wrong word. It implies a unilateral action to demote your site. The phrase I use in terms of diagnosis is "below expected rank". If you add up all the factors and see where you appear, is it significantly below expectation? How did you arrive at that expectation?
Otherwise, you're experiencing symptoms explained by Google's approach to indexing and ranking. Google only show what's required, so 5 results are what's required to satisfy the user's search. When you start digging deeper, you start to trigger Google's querying of deeper datasets. It's the same reason your result numbers are so different.
The first question is, at what point did Google decide your listing wasn't relevant? Was it during indexing, or did you get re-ranked?
Move away from penalty, and towards relevance.
Incidentally, be cautious about evaluating Google by clicking links in search results. Filters and options (and thus relevance) change with almost every option, and so this can lead you in the wrong direction. Check your Google result URLs.