Welcome to WebmasterWorld Guest from 50.19.190.144

Forum Moderators: goodroi

Message Too Old, No Replies

Robots text to block all but index page of a forum

     
10:41 pm on Dec 21, 2011 (gmt 0)

Preferred Member

joined:Mar 22, 2011
posts:395
votes: 0


I'm trying to generate an robot text file to disallow all SEs on all pages of forum except the index page.

So I want to allow it to index:

mysite.com/forum/


But anything else in the forum directory and subdirectories, I don't want indexed.

mysite.com/forum/$anythingelse$

I remember a command with a question mark in it that worked in the past. But I can't find any guidance on that.


Does anyone know how to do that?
12:25 am on Dec 22, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


What directly follows /forum/ in the URLs to be blocked?

Is it any letter from A to Z or selected letters or certain digits?
1:06 am on Dec 22, 2011 (gmt 0)

Preferred Member

joined:Mar 22, 2011
posts:395
votes: 0


All are letters, no numbers at all.

Thanks for the help.
2:53 am on Dec 22, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 31, 2005
posts:1108
votes: 0


You could do the following
Disallow: /forum/a
Disallow: /forum/b
Disallow: /forum/c
etc.
Disallow: /forum/z


The question mark and other wild cards are not supported by all bots.
7:35 am on Dec 22, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


Is it any letter from A to Z or selected letters?

Are they all the same case?
11:31 am on Dec 22, 2011 (gmt 0)

Preferred Member

joined:Mar 22, 2011
posts:395
votes: 0


It's a number of different letters maybe 10.

They are all lower case.

Thanks a lot guys.
11:41 am on Dec 22, 2011 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:6130
votes: 277


Allow forum/index.html
Disallow forum/

Works for me in real life.

mysite.com/ is not required

Order is important.
12:34 pm on Dec 22, 2011 (gmt 0)

Preferred Member

joined:Mar 22, 2011
posts:395
votes: 0


Thanks tangor. You guys rock.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members