Welcome to WebmasterWorld Guest from 54.145.44.134

Forum Moderators: goodroi

Message Too Old, No Replies

Disallow pages in a subdirectory

Basic help please

     

pjuk

9:49 am on Jun 14, 2004 (gmt 0)

10+ Year Member



Hi - I'm new to the forum and new to robot.txt files, this is maybe a stupid question but I want to be sure before I upload the robots.txt file.

I want to disallow robots from three pages that 'live' in a sub folder called 'pages' I have set up the robots.txt file as follows-

User-agent: *

Disallow: /pages/paypalcanceled.htm
Disallow: /pages/paypalconfirmed.htm
Disallow: /pages/booking form.htm

Is this correct? I do not want to disallow all files in 'pages' only the listed ones.
Also I have saved the robots.txt file to the root directory as a txt. file is this correct?

Thanks for any advice

pjuk

goodroi

7:02 pm on Jun 24, 2004 (gmt 0)

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Yes that is correct. Make sure to use a robots.txt validator. I've caught many typos that way. Also as an extra precaution you can place noindex, nofollow tags within the html of those pages.

jdMorgan

8:40 pm on Jun 24, 2004 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



pj,

The code posted above may not work as expected, because the Standard for Robots Exclusion [robotstxt.org] specifies that a blank line is to be interpreted as an end-of-record indicator. Delete the blank line, and use:


User-agent: *
Disallow: /pages/paypalcanceled.htm
Disallow: /pages/paypalconfirmed.htm
Disallow: /pages/booking form.htm

As goodroi suggests, validate your robots.txt file [searchengineworld.com] before using it.

Jim

 

Featured Threads

Hot Threads This Week

Hot Threads This Month