There are probably millions of sites never designed to display well on the small screen of a mobile phone. Heavy content informative sites are a case in point. Not surprisingly, the Page Speed Insights in GWT for mobile shows a poor result, at least with one of my sites.
Is a site with a poor mobile Page Speed result likely to be ranked lower in Google's SERP's because it has been assessed as having a weaker user experience?
I'm looking into this as well. I agree with netmeg. If google is showing you results, in webmaster tools, adsense accounts or analytic s for mobile users it matters.
In the end its going to come down to user experiences and for example on my main site, I have seen as high as 50% high end mobiles and tablets. so the fact my site doesn't display well for them will impact their experience, so if they leave faster, if will and probably / does have a negative impact for a site, even if you have the best page out there. As it will causes a higher then expected bounce rate.
If you are like me you missed the whole m.domain.com era, and are now in the new responsive website age. Work on the pages that will have the biggest impact on your site first. And learn how to hide, certain bandwidth heavy elements for a site, or at least load them last, so your users see something on your pages.