Welcome to WebmasterWorld Guest from 54.160.221.82

Forum Moderators: goodroi

Message Too Old, No Replies

Disallow pages in a subdirectory

Basic help please

     

pjuk

9:49 am on Jun 14, 2004 (gmt 0)

Inactive Member
Account Expired

 
 


Hi - I'm new to the forum and new to robot.txt files, this is maybe a stupid question but I want to be sure before I upload the robots.txt file.

I want to disallow robots from three pages that 'live' in a sub folder called 'pages' I have set up the robots.txt file as follows-

User-agent: *

Disallow: /pages/paypalcanceled.htm
Disallow: /pages/paypalconfirmed.htm
Disallow: /pages/booking form.htm

Is this correct? I do not want to disallow all files in 'pages' only the listed ones.
Also I have saved the robots.txt file to the root directory as a txt. file is this correct?

Thanks for any advice

pjuk

7:02 pm on June 24, 2004 (gmt 0)

Administrator from US 

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 21, 2004
posts:3080
votes: 67


Yes that is correct. Make sure to use a robots.txt validator. I've caught many typos that way. Also as an extra precaution you can place noindex, nofollow tags within the html of those pages.
8:40 pm on June 24, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


pj,

The code posted above may not work as expected, because the Standard for Robots Exclusion [robotstxt.org] specifies that a blank line is to be interpreted as an end-of-record indicator. Delete the blank line, and use:


User-agent: *
Disallow: /pages/paypalcanceled.htm
Disallow: /pages/paypalconfirmed.htm
Disallow: /pages/booking form.htm

As goodroi suggests, validate your robots.txt file [searchengineworld.com] before using it.

Jim

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members