tedster - 11:44 pm on Nov 24, 2010 (gmt 0)
This is a pretty cool reference - the exact information about Google's current handling of robots directives:
robots.txt specifications [code.google.com]
robots meta tags and x-robots directives [code.google.com]
I've already learned something new. Google will look for and obey an FTP robots.txt file located at ftp://example.com/robots.txt
And here's the cover page for the entire robots collection [code.google.com].