homepage Welcome to WebmasterWorld Guest from 54.204.215.209
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe and Support WebmasterWorld
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Confused about robots.txt
need a little help
Yamagotchi




msg:1527299
 6:54 am on Oct 21, 2002 (gmt 0)

I am trying to setup a robots.txt on my site to disallow a file.

The file is named dex.php

so I have setup a robots.txt as follows:

User-agent: *
Disallow: /dex.php

Will this make is so spiders do not crawl 'dex.php' and dex.php?s=12345?

Also, will this affect inDEX.php?

TIA for any help.

 

andreasfriedrich




msg:1527300
 3:19 pm on Oct 21, 2002 (gmt 0)

The value of this field [Disallow] specifies a partial URL that is not to be visited. This can be a full path, or a partial path; any URL that starts with this value will not be retrieved

A Standard for Robot Exclusion [robotstxt.org] - my emphasis

'/dex.php' and '/dex.php?s=12345' are disallowed, 'inDEX.php' is not since it does not start with /dex.php.

Andreas

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved