Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

excluding cgi-bin

is this right

2:39 am on Nov 10, 2004 (gmt 0)

Senior Member

joined:Nov 29, 2003
votes: 0

I know this is way basic but I need to be absolutely sure I get it right. I don't want any robots to spider my cgi-bin directory. Is the following correct, if placed in the directory where my web pages live (httpdocs)?

User-agent: *
Disallow: /cgi-bin/

2:59 am on Nov 10, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 7, 2003
votes: 0

Yes it is the correct format

A robots.txt tutorial and validator