robots.txt file

[ktd121]

After looking at my site root I did not see a robots.txt file. What exactly does this file do and how do you make one? What do you need to put into one and do you need to have one?

[PatrickDeese]

robots.txt is used to forbid search engine crawlers access to certain files & directories on your site.

For instance if you didn't want to have your visitor stats crawled by the search engines, perhaps you would forbid the directory stats/ or where ever your stats page lives.

try [robotstxt.org...] or maybe

[webmasterworld.com...]

[Woz]

>robots.txt is used to forbid search engine crawlers access to certain files & directories on your site.

This is assuming they follow robots.txt protocols. Don't rely on robots.txt file alone to protect sensitive information of temporary folders/files.

Woz