Welcome to WebmasterWorld Guest from 54.205.74.11

Forum Moderators: goodroi

robots.txt file

   
2:16 am on Jul 4, 2003 (gmt 0)

10+ Year Member



After looking at my site root I did not see a robots.txt file. What exactly does this file do and how do you make one? What do you need to put into one and do you need to have one?
2:23 am on Jul 4, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



robots.txt is used to forbid search engine crawlers access to certain files & directories on your site.

For instance if you didn't want to have your visitor stats crawled by the search engines, perhaps you would forbid the directory stats/ or where ever your stats page lives.

try [robotstxt.org...] or maybe

[webmasterworld.com...]

Woz

3:14 am on Jul 4, 2003 (gmt 0)

WebmasterWorld Senior Member woz is a WebmasterWorld Top Contributor of All Time 10+ Year Member



>robots.txt is used to forbid search engine crawlers access to certain files & directories on your site.

This is assuming they follow robots.txt protocols. Don't rely on robots.txt file alone to protect sensitive information of temporary folders/files.

Woz

 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month