Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Confused about robots.txt

need a little help



6:54 am on Oct 21, 2002 (gmt 0)

10+ Year Member

I am trying to setup a robots.txt on my site to disallow a file.

The file is named dex.php

so I have setup a robots.txt as follows:

User-agent: *
Disallow: /dex.php

Will this make is so spiders do not crawl 'dex.php' and dex.php?s=12345?

Also, will this affect inDEX.php?

TIA for any help.


3:19 pm on Oct 21, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

The value of this field [Disallow] specifies a partial URL that is not to be visited. This can be a full path, or a partial path; any URL that starts with this value will not be retrieved

A Standard for Robot Exclusion [robotstxt.org] - my emphasis

'/dex.php' and '/dex.php?s=12345' are disallowed, 'inDEX.php' is not since it does not start with /dex.php.



Featured Threads

Hot Threads This Week

Hot Threads This Month