homepage Welcome to WebmasterWorld Guest from 54.161.192.61
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots.txt exclusion question
Robots.txt exclude all but one directory?
olof_bailey

10+ Year Member



 
Msg#: 4 posted 8:48 am on Sep 10, 2003 (gmt 0)

Right basically I want to exclude an entire site except for one directory (a forum). The problem is that this forum directory resides in the root of the site (with all of the active pages I need to exclude)

Probably a silly question but is there any way of doing this without excluding all of the files individually?

Cheers Guys

Rich

 

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4 posted 1:19 pm on Sep 10, 2003 (gmt 0)

Rich,

The simple answer is "No." There is only a "Disallow" directive in the robots exclusion standard.

The complex answer is that you can move your pages into separate directory branches, depending on whether you want them spidered/indexed, and set up robots.txt to disallow all but one directory branch.

You could also use mod_rewrite or a similar approach to make it look as if the directory structure had been re-organized as above, but leave the pages where they are.

Or, you could use a script to generate robots.txt, allowing only the desired pages, and saving you the work of maintaining it.

Depending on your site layout, these approaches might vary from easy to horribly complex or inefficient.

Jim

olof_bailey

10+ Year Member



 
Msg#: 4 posted 1:33 pm on Sep 10, 2003 (gmt 0)

ouch....

olof_bailey

10+ Year Member



 
Msg#: 4 posted 8:52 pm on Sep 20, 2003 (gmt 0)

anymore for anymore?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved