homepage Welcome to WebmasterWorld Guest from 54.227.20.250
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Disallow pages in a subdirectory
Basic help please
pjuk

10+ Year Member



 
Msg#: 406 posted 9:49 am on Jun 14, 2004 (gmt 0)

Hi - I'm new to the forum and new to robot.txt files, this is maybe a stupid question but I want to be sure before I upload the robots.txt file.

I want to disallow robots from three pages that 'live' in a sub folder called 'pages' I have set up the robots.txt file as follows-

User-agent: *

Disallow: /pages/paypalcanceled.htm
Disallow: /pages/paypalconfirmed.htm
Disallow: /pages/booking form.htm

Is this correct? I do not want to disallow all files in 'pages' only the listed ones.
Also I have saved the robots.txt file to the root directory as a txt. file is this correct?

Thanks for any advice

pjuk

 

goodroi

WebmasterWorld Administrator goodroi us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 406 posted 7:02 pm on Jun 24, 2004 (gmt 0)

Yes that is correct. Make sure to use a robots.txt validator. I've caught many typos that way. Also as an extra precaution you can place noindex, nofollow tags within the html of those pages.

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 406 posted 8:40 pm on Jun 24, 2004 (gmt 0)

pj,

The code posted above may not work as expected, because the Standard for Robots Exclusion [robotstxt.org] specifies that a blank line is to be interpreted as an end-of-record indicator. Delete the blank line, and use:

User-agent: *
Disallow: /pages/paypalcanceled.htm
Disallow: /pages/paypalconfirmed.htm
Disallow: /pages/booking form.htm

As goodroi suggests, validate your robots.txt file [searchengineworld.com] before using it.

Jim

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved