I understand the difference. Many people talk about a "penalty," but what they really mean is that a factor that once helped them rank no longer helps them rank. For the most part people hit by Panda aren't being "penalized" they are just no longer getting ranking credit for something that once helped them.
Or people will talk about a duplicate content penalty, which is just wrong. There is no duplicate content penalty.
Duplicate content, for example, prevents you from ranking because Google is splitting authority and can't figure out your structure. But Google will *try* to solve your dupe content issues for you if it can.
With the mobile issue, though, this is different. Google is analyzing your site and making an active effort to remove you from the search results.
No, it isn't a manual action or a webspam violation, but to me it's more like a penalty than it is like the dupe content or even thin content problem.
Anyway, it's mostly just semantics, but to me the main thing is the distinction between
1. ranking poorly because you are failing to obtain positive factors on your behalf
2. ranking poorly because Google is punishing you in the rankings because of negative factors.
Dupe content, most rankings drops due to Penguin and Panda are #1
The mobile issue is #2.
[update]Having said that and having that impression from the dire email from Google, today's Google Webmaster Central presents it more like a dupe content/Panda/Penguin thing where it's more a matter of failing to send a positive signal.
In any case, it is a matter of concern that will have a significant impact on your rankings.