Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: goodroi
The file is named dex.php
so I have setup a robots.txt as follows:
Will this make is so spiders do not crawl 'dex.php' and dex.php?s=12345?
Also, will this affect inDEX.php?
TIA for any help.
The value of this field [Disallow] specifies a partial URL that is not to be visited. This can be a full path, or a partial path; any URL that starts with this value will not be retrieved
A Standard for Robot Exclusion [robotstxt.org] - my emphasis
'/dex.php' and '/dex.php?s=12345' are disallowed, 'inDEX.php' is not since it does not start with /dex.php.