I'm not sure of which is the most efficient but considering mod_security is yet another layer of add-ons, I'd assume it would be slightly less efficient just having it there in the first place.
Overall REMOTE_HOST is grossly inefficient as it forces a reverse DNS lookup which can burn a lot of time opposed to simply blocking by IP range.
I don't mind reverse DNS look ups if they're cached, which I do in my code but I can't voich for REMOTE_HOST, so it doesn't repetitively do it every time the server encounters the same IP in a short time period.
This old thread is enough to get you started.
Yeah, but that's blacklisting and the number of user agents to block is in the thousands now and the amount of linear processing added to every Apache process is ridiculously inefficient.
Do the same basic code but whitelisting and it's short, efficient and permanently effective, you're good to go.