homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Robots.txt exclusion question
Robots.txt exclude all but one directory?

 8:48 am on Sep 10, 2003 (gmt 0)

Right basically I want to exclude an entire site except for one directory (a forum). The problem is that this forum directory resides in the root of the site (with all of the active pages I need to exclude)

Probably a silly question but is there any way of doing this without excluding all of the files individually?

Cheers Guys




 1:19 pm on Sep 10, 2003 (gmt 0)


The simple answer is "No." There is only a "Disallow" directive in the robots exclusion standard.

The complex answer is that you can move your pages into separate directory branches, depending on whether you want them spidered/indexed, and set up robots.txt to disallow all but one directory branch.

You could also use mod_rewrite or a similar approach to make it look as if the directory structure had been re-organized as above, but leave the pages where they are.

Or, you could use a script to generate robots.txt, allowing only the desired pages, and saving you the work of maintaining it.

Depending on your site layout, these approaches might vary from easy to horribly complex or inefficient.



 1:33 pm on Sep 10, 2003 (gmt 0)



 8:52 pm on Sep 20, 2003 (gmt 0)

anymore for anymore?

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved