lucy24 - 1:28 am on May 11, 2013 (gmt 0)
Why are you futzing around with robots.txt? Bad bots probably don't read it and certainly don't obey it; the only option is to block 'em at the source. Lotsa ways to do this depending on server and personal preference.
"allow" unlike "disallow" is not a cast-in-stone part of the robots exclusion protocol, so robots can legitimately ignore it and still retain their halos. (The same applies to "Crawl-Delay".)
Version number should have no effect. I asked about this recently and someone-- phranque, I think-- pointed to a passage in The Rules that says "user-agent" should be interpreted broadly. If you're not sure a rule applies to you, assume it does.
If you have a big, high-traffic site you can start doing fancy things like serving each robot a custom robots.txt that names only itself, so the robot can't sneak off, change clothes and come back disguised as the googlebot to get wider access. Not that this would do the robot any good: It's more likely to net it a swift 403. For ordinary mortals a custom robots.txt isn't worth the trouble.