A lower QS will make your ads more expensive, but there's no reason to ditch the keywords unless they're actually unprofitable - provided you can track conversion on the keyword level of course.
I have heard that historical CTR across your account does impact your quality score, so that might be a factor. Provided you have decent relevancy, good ad copy, and a good landing page there's no real reason why your quality score should be that low. But you do have to have those three things, so don't forget about your landing pages and ad copy (they can always be improved).
It may be worth generating a search query report and seeing if any of your keywords are matching to queries which are irrelevant or unqualified. Negativing those queries out will improve your CTR and thus your ads' QS.