I am hesitant to remove them outright because they may have links in the past and removing them would cause 404 and potential loss of past link juice.
? If you've removed them on purpose, adding the appropriate redirect code, it's not a 404. It's a 410. This makes no apparent difference to No.-2-We-Try-Harder, but it's very noticeable with Google.
But if there are active desirable links to the pages, you wouldn't want to remove them anyway. Dump the ones that don't have links to them-- and then only if they're creating unwanted clutter merely by existing. Not likely, unless your internal navigation is based on mandatory links: if it's there, it must be linked-to.