Welcome to WebmasterWorld Guest from 54.144.110.101

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt exclusion question

Robots.txt exclude all but one directory?

     
8:48 am on Sep 10, 2003 (gmt 0)

New User

10+ Year Member

joined:July 7, 2003
posts:12
votes: 0


Right basically I want to exclude an entire site except for one directory (a forum). The problem is that this forum directory resides in the root of the site (with all of the active pages I need to exclude)

Probably a silly question but is there any way of doing this without excluding all of the files individually?

Cheers Guys

Rich

1:19 pm on Sept 10, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


Rich,

The simple answer is "No." There is only a "Disallow" directive in the robots exclusion standard.

The complex answer is that you can move your pages into separate directory branches, depending on whether you want them spidered/indexed, and set up robots.txt to disallow all but one directory branch.

You could also use mod_rewrite or a similar approach to make it look as if the directory structure had been re-organized as above, but leave the pages where they are.

Or, you could use a script to generate robots.txt, allowing only the desired pages, and saving you the work of maintaining it.

Depending on your site layout, these approaches might vary from easy to horribly complex or inefficient.

Jim

1:33 pm on Sept 10, 2003 (gmt 0)

New User

10+ Year Member

joined:July 7, 2003
posts:12
votes: 0


ouch....
8:52 pm on Sept 20, 2003 (gmt 0)

New User

10+ Year Member

joined:July 7, 2003
posts:12
votes: 0


anymore for anymore?
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members