| Welcome to WebmasterWorld Guest from 22.214.171.124 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Pubcon Platinum Sponsor 2014|
|Confused about robots.txt|
need a little help
| 6:54 am on Oct 21, 2002 (gmt 0)|
I am trying to setup a robots.txt on my site to disallow a file.
The file is named dex.php
so I have setup a robots.txt as follows:
Will this make is so spiders do not crawl 'dex.php' and dex.php?s=12345?
Also, will this affect inDEX.php?
TIA for any help.
| 3:19 pm on Oct 21, 2002 (gmt 0)|
|The value of this field [Disallow] specifies a partial URL that is not to be visited. This can be a full path, or a partial path; any URL that starts with this value will not be retrieved |
A Standard for Robot Exclusion [robotstxt.org] - my emphasis
'/dex.php' and '/dex.php?s=12345' are disallowed, 'inDEX.php' is not since it does not start with /dex.php.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved