I'm not sure from your post if you've split out exact match keywords. If not, you want to try further segment your top performing keywords to get the exact matches in separate ad groups.
You are correct in your assumption re: specific negatives on the other AdGroups. If you have top performing exact match keywords which are being matched by other broad or phrase keywords then add the exact negative to the other ad groups that are matching. It shouldn't be too hard to with a bit of excel wizardry and the desktop editor.
Alternatively, you could put all your broad/phrase match keywords in separate campaigns and put in campaign negatives.