I am right in thinking aren't I that robots.txt and the robots meta tags can work together? So you can have:
and then control what `abbabot` can and cannot index via meta tags, which would be these primarily:
<META name="ROBOTS" content="all">
<META name="ROBOTS" content="none">
There are reasons I don't want to use robots.txt, the site in question adds new sections/channels on a daily basis, many of which we don't want spidered, some we do, whilst the URL structure is such that even with using say the Google extensions to robots.txt we'd end up with a very, very large robots.txt file which would be unmanageable on a daily basis. With this we can use the CMS (Content Management System) to control the state of meta tags when the pages are created. Well it sounds like a plan anyway...
I see that at some point the W3C discussed putting user agents in the meta tag standard but didn't...