Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt exclusion question

Robots.txt exclude all but one directory?

8:48 am on Sep 10, 2003 (gmt 0)

10+ Year Member

Right basically I want to exclude an entire site except for one directory (a forum). The problem is that this forum directory resides in the root of the site (with all of the active pages I need to exclude)

Probably a silly question but is there any way of doing this without excluding all of the files individually?

Cheers Guys


1:19 pm on Sep 10, 2003 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member


The simple answer is "No." There is only a "Disallow" directive in the robots exclusion standard.

The complex answer is that you can move your pages into separate directory branches, depending on whether you want them spidered/indexed, and set up robots.txt to disallow all but one directory branch.

You could also use mod_rewrite or a similar approach to make it look as if the directory structure had been re-organized as above, but leave the pages where they are.

Or, you could use a script to generate robots.txt, allowing only the desired pages, and saving you the work of maintaining it.

Depending on your site layout, these approaches might vary from easy to horribly complex or inefficient.


1:33 pm on Sep 10, 2003 (gmt 0)

10+ Year Member

8:52 pm on Sep 20, 2003 (gmt 0)

10+ Year Member

anymore for anymore?

Featured Threads

Hot Threads This Week

Hot Threads This Month