Welcome to WebmasterWorld Guest from 54.234.38.8

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt file

     
2:16 am on Jul 4, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:June 22, 2003
posts:45
votes: 0


After looking at my site root I did not see a robots.txt file. What exactly does this file do and how do you make one? What do you need to put into one and do you need to have one?
2:23 am on July 4, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 6, 2003
posts:2523
votes: 0


robots.txt is used to forbid search engine crawlers access to certain files & directories on your site.

For instance if you didn't want to have your visitor stats crawled by the search engines, perhaps you would forbid the directory stats/ or where ever your stats page lives.

try [robotstxt.org...] or maybe

[webmasterworld.com...]

Woz

3:14 am on July 4, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member woz is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 13, 2000
posts:4823
votes: 0


>robots.txt is used to forbid search engine crawlers access to certain files & directories on your site.

This is assuming they follow robots.txt protocols. Don't rely on robots.txt file alone to protect sensitive information of temporary folders/files.

Woz