Forum Moderators: Robert Charlton & goodroi
Thanks :)
the minimal robots.txt file for excluding google search from indexing a directory named "go-away" would be:
User-agent: googlebot
Disallow: /go-away/
you can also use robots meta tags such as noindex and nofollow to control googlebot behavior on a per-page basis.
you can get more information in the robots.txt forum [webmasterworld.com].
check out the Charter link at the top of the forum list for references.