It does, but at the end of the day, there's still only ten spots on page one, and gazillions of people competing for them. Including some with some pretty big pockets, and you *are* in a competitive niche.
Do you get a lot of repeat visitors? For a site such as yours, I'd be working on loyalty, making sure people come back often and share your stuff. Running an email newsletter, so every now and then you can remind people to come back to your site.
If you haven't done any linkbuilding, then don't worry about the links. I doubt they are causing your problems.
Google just released a watered down version of their Search Quality Rater's Guide - you could take a look at that and see if you can find any hints in there on how to reasonably gauge your site vs the competition.
(that's a 43 page PDF)