Welcome to WebmasterWorld Guest from 23.20.18.183

Forum Moderators: goodroi

Message Too Old, No Replies

excluding cgi-bin

is this right

     
2:39 am on Nov 10, 2004 (gmt 0)

Senior Member

joined:Nov 29, 2003
posts:875
votes: 0


I know this is way basic but I need to be absolutely sure I get it right. I don't want any robots to spider my cgi-bin directory. Is the following correct, if placed in the directory where my web pages live (httpdocs)?

User-agent: *
Disallow: /cgi-bin/

2:59 am on Nov 10, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 7, 2003
posts:1179
votes: 0


Yes it is the correct format

A robots.txt tutorial and validator
[searchengineworld.com...]
[searchengineworld.com...]