That's very possible, aristotle. The user interaction on the site isn't where I want it to be, but it's something I'm analyzing closely and actively working on.
I am to the point where I think I need to re-assess my target keywords. It's hard to explain without getting into specifics of the site, but let's just say that two very different audiences with two very different intents search for the same keywords with roughly equal volume. My gut tells me that this is a major factor as far as hurting time on site and increasing bounces at the aggregate level. So penalty or not, it's something I need to improve.
There is another variation that's more audience specific, but less specific to my content (an annoying problem). I think I need to pivot and go after that one instead.
Edit to say that shifting keyword focus is something I've been considering since before this drop in traffic.