wilderness - 1:56 pm on May 31, 2013 (gmt 0)
I was using Deny from 168. xxx etc but I was told by the server that it took uo too much processor time
A single denial of a Class A causes no such server load.
A server load, however could occur caused by loops and malfunctioning Error Documens.
I've an approximate dozen Class B's in the 168 denied, otherwise the A is open access. May have some 168's in custom rules.
I doubt some of those you have marked as working actually were working.
I agree with Key_master.
In order for most of those UA's to function you'll need to remove the "begins with" anchor.
A few of those bots are compliant and will honor robots.txt.
You don't seemed to have addressed the major SE's previews, which don't fall into the category of regular bot and masquerade themselves as standard browsers.
MSN-Bing potentially become real real crawling pests if left unrestricted.
Do your site (s) also contain images and/or other types of media files that are not restricted for bot crawling?