Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- robot.txt syntax

korkus2000 - 2:45 pm on Nov 18, 2002 (gmt 0)


Here is a great tutorial on robots.txt
[searchengineworld.com...]

This bars all robots fro the cgi-bin folder.

User-agent: *
Disallow: /cgi-bin/

Place it on the site's root.

You can also validate it at [searchengineworld.com...]

Thread source:: http://www.webmasterworld.com/robots_txt/126.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com