also be aware that when you are using nonstandard robots.txt syntax - bad things can happen. Never use nonstandard syntax with wildcard agent names.
That will disallow the entire site on some bots.
There is no "allow" syntax and we have documented Slurp misinterpreting it in the past as a disallow line which ultimately removed the entire site from Yahoo. That behavior has been changed, but it is a clear cut case of what can go wrong when using nonstandard (improper) syntax in robots.txt.