homepage Welcome to WebmasterWorld Guest from 54.167.96.124
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
robots.txt expert question
Is it possible for robots.txt to disallow all but one page?
DanielDiGriz




msg:1526021
 7:58 pm on Aug 16, 2003 (gmt 0)

In robots.txt I have:

User-agent: *
Disallow: /

What I want, instead, is to disallow all but one page, *without having to specifically disallow each page on the site* (pages are added constantly). Am I correct that robots.txt provides no way to do this, since the protocol is to disallow rather than allow?

 

jatar_k




msg:1526022
 9:02 pm on Aug 16, 2003 (gmt 0)

There is no allow, an option might be to disallow everything but that one page by doing them all explicitly.

User-agent: *
Disallow: /index.html
Disallow: /pages/
Disallow: /img/
Disallow: /otherpage.html

then have the page you want to allow in the root and disallow everything but.

jdMorgan




msg:1526023
 9:34 pm on Aug 16, 2003 (gmt 0)

DanielDiGriz,

Welcome to WebmasterWorld [webmasterworld.com]!

It can be helpful to split files which are allowed from those which are disallowed into different subdirectories. In this way, you can disallow a subdirectory path, and add content to it at will. Since the whole subdirectory is disallowed, you won't need to change robots.txt.

Jim

DanielDiGriz




msg:1526024
 11:38 pm on Aug 16, 2003 (gmt 0)

Explicitly is not an option, but I like the subdirectory idea. Thank you for the responses. :)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved