Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Why would I use a Robots.txt file?

10:15 pm on Jan 11, 2001 (gmt 0)

Preferred Member

10+ Year Member

joined:Nov 25, 2000
votes: 0

Is there a good set of rules to follow as far as keeping spidey off certain pages???
IP delivery, cgi, etc.


4:20 am on Jan 12, 2001 (gmt 0)

Inactive Member
Account Expired


What exactly is the robots.txt file used for?


12:36 am on Jan 13, 2001 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 19, 2000
votes: 0

The robots.txt file (and the robots) META tag are used to
tell robots what they should and should not access.

Most robots follow the robots exclusion standard, which can be found at

The idea of limiting a robots crawling is that you might not want all pages indexed (any inexed page can be an entrypoint to your site). You wouldn't want to start your visit at the feedback page, or perhaps you dont want robots to index your pdf docs which you keep in a certain dir. Or there might be a section that's password protected, eg.

Just ideas

3:42 pm on Jan 13, 2001 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
votes: 16

Robots is good for blocking off high profile directories. If it specific pages you want to block, I'd use a meta tag No Index tag - far more effective. Engines (like Google) will check for a robots once a month, whereas the meta tag is read every page read.
5:04 pm on Jan 13, 2001 (gmt 0)

Preferred Member

10+ Year Member

joined:Nov 25, 2000
votes: 0

Thanks Brett,
High profile meaning what ?

Forgive me..sometimes I'm retarded.

11:58 pm on Jan 13, 2001 (gmt 0)

Senior Member

joined:June 27, 2000
votes: 0


We used a robots.txt file to keep the privacy policy pages from being the first listed in the search results when the site was spidered. It was embarrasing to type in the keyword for the site, and the first listing was "This our dry legal privacy policy". When people saw that page pop up, they wouldn't even click a nav link on the page to go somewhere else, they would just hit back and avoid the site all together.

*shrug* That's why WE used it.


12:40 am on Jan 14, 2001 (gmt 0)

Preferred Member

10+ Year Member

joined:Nov 25, 2000
votes: 0

Thanx G-

I guess I better rename my IP delivery folder something other than "Super-Secret Search Engine Redirection Script" before I write that robots.txt file, huh?