My point is more should I take it away as rather than generating any traffic for us it is just making our CTR bad, thus would I be right in thinking then that Google would class our Ad as not so relevant on all the other terms in that campaign. Therefore dropping our Ad down the list.
If you have a keyword that is not giving you good results (as defined by you), then, by all means remove it. However, don't worry about a keyword with a poor CTR pulling an entire Ad Group down in position. Position is determined on a keyword-by-keyword basis - not per Ad Group.