Welcome to WebmasterWorld Guest from 23.20.6.115

Forum Moderators: goodroi

Message Too Old, No Replies

Confused about robots.txt

need a little help

     
6:54 am on Oct 21, 2002 (gmt 0)

New User

10+ Year Member

joined:Oct 15, 2002
posts:5
votes: 0


I am trying to setup a robots.txt on my site to disallow a file.

The file is named dex.php

so I have setup a robots.txt as follows:

User-agent: *
Disallow: /dex.php

Will this make is so spiders do not crawl 'dex.php' and dex.php?s=12345?

Also, will this affect inDEX.php?

TIA for any help.

3:19 pm on Oct 21, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 22, 2002
posts:1782
votes: 0


The value of this field [Disallow] specifies a partial URL that is not to be visited. This can be a full path, or a partial path; any URL that starts with this value will not be retrieved

A Standard for Robot Exclusion [robotstxt.org] - my emphasis

'/dex.php' and '/dex.php?s=12345' are disallowed, 'inDEX.php' is not since it does not start with /dex.php.

Andreas