homepage Welcome to WebmasterWorld Guest from 23.22.212.158
register, login, search, subscribe, help, library, PubCon, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library : Charter : Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots text to block all but index page of a forum
Pjman




msg:4400296		 10:41 pm on Dec 21, 2011 (gmt 0)

I'm trying to generate an robot text file to disallow all SEs on all pages of forum except the index page.

So I want to allow it to index:

mysite.com/forum/


But anything else in the forum directory and subdirectories, I don't want indexed.

mysite.com/forum/$anythingelse$

I remember a command with a question mark in it that worked in the past. But I can't find any guidance on that.


Does anyone know how to do that?

 

g1smd




msg:4400333		 12:25 am on Dec 22, 2011 (gmt 0)

What directly follows /forum/ in the URLs to be blocked?

Is it any letter from A to Z or selected letters or certain digits?
Pjman




msg:4400350		 1:06 am on Dec 22, 2011 (gmt 0)

All are letters, no numbers at all.

Thanks for the help.
Dijkgraaf




msg:4400365		 2:53 am on Dec 22, 2011 (gmt 0)

You could do the following
Disallow: /forum/a
Disallow: /forum/b
Disallow: /forum/c
etc.
Disallow: /forum/z


The question mark and other wild cards are not supported by all bots.
g1smd




msg:4400403		 7:35 am on Dec 22, 2011 (gmt 0)

Is it any letter from A to Z or selected letters?

Are they all the same case?
Pjman




msg:4400465		 11:31 am on Dec 22, 2011 (gmt 0)

It's a number of different letters maybe 10.

They are all lower case.

Thanks a lot guys.
tangor




msg:4400468		 11:41 am on Dec 22, 2011 (gmt 0)

Allow forum/index.html
Disallow forum/

Works for me in real life.

mysite.com/ is not required

Order is important.
Pjman




msg:4400484		 12:34 pm on Dec 22, 2011 (gmt 0)

Thanks tangor. You guys rock.
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
WebmasterWorld ® and PubCon ® are a Registered Trademarks of Pubcon Inc.
© Pubcon Inc. 1996-2012 all rights reserved