Welcome to WebmasterWorld Guest from 23.21.38.201

Forum Moderators: goodroi

Confused about robots.txt

need a little help

   
6:54 am on Oct 21, 2002 (gmt 0)

10+ Year Member



I am trying to setup a robots.txt on my site to disallow a file.

The file is named dex.php

so I have setup a robots.txt as follows:

User-agent: *
Disallow: /dex.php

Will this make is so spiders do not crawl 'dex.php' and dex.php?s=12345?

Also, will this affect inDEX.php?

TIA for any help.

3:19 pm on Oct 21, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The value of this field [Disallow] specifies a partial URL that is not to be visited. This can be a full path, or a partial path; any URL that starts with this value will not be retrieved

A Standard for Robot Exclusion [robotstxt.org] - my emphasis

'/dex.php' and '/dex.php?s=12345' are disallowed, 'inDEX.php' is not since it does not start with /dex.php.

Andreas

 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month