Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

excluding cgi-bin

is this right



2:39 am on Nov 10, 2004 (gmt 0)

I know this is way basic but I need to be absolutely sure I get it right. I don't want any robots to spider my cgi-bin directory. Is the following correct, if placed in the directory where my web pages live (httpdocs)?

User-agent: *
Disallow: /cgi-bin/


2:59 am on Nov 10, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Yes it is the correct format

A robots.txt tutorial and validator


Featured Threads

Hot Threads This Week

Hot Threads This Month