Forum Moderators: goodroi
A few examples of what you would put in your robots.txt file
To exclude all robots from parts of the server
User-agent: *
Disallow: /cgi-bin/
Disallow: /misc/sitestats/
This indicates that nothing is disallowed and the spider can follow all links
User-agent: *
Disallow:
To allow a single robot complete access and exclude all others
User-agent: Googlebot
Disallow:
User-agent: *
Disallow: /
This would prevent your entire web site from being indexed
User-agent: *
Disallow: /
hope this helps
ncw164x