homepage Welcome to WebmasterWorld Guest from 54.226.191.80
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots text to block all but index page of a forum
Pjman




msg:4400296
 10:41 pm on Dec 21, 2011 (gmt 0)

I'm trying to generate an robot text file to disallow all SEs on all pages of forum except the index page.

So I want to allow it to index:

mysite.com/forum/


But anything else in the forum directory and subdirectories, I don't want indexed.

mysite.com/forum/$anythingelse$

I remember a command with a question mark in it that worked in the past. But I can't find any guidance on that.


Does anyone know how to do that?

 

g1smd




msg:4400333
 12:25 am on Dec 22, 2011 (gmt 0)

What directly follows /forum/ in the URLs to be blocked?

Is it any letter from A to Z or selected letters or certain digits?

Pjman




msg:4400350
 1:06 am on Dec 22, 2011 (gmt 0)

All are letters, no numbers at all.

Thanks for the help.

Dijkgraaf




msg:4400365
 2:53 am on Dec 22, 2011 (gmt 0)

You could do the following
Disallow: /forum/a
Disallow: /forum/b
Disallow: /forum/c
etc.
Disallow: /forum/z


The question mark and other wild cards are not supported by all bots.

g1smd




msg:4400403
 7:35 am on Dec 22, 2011 (gmt 0)

Is it any letter from A to Z or selected letters?

Are they all the same case?

Pjman




msg:4400465
 11:31 am on Dec 22, 2011 (gmt 0)

It's a number of different letters maybe 10.

They are all lower case.

Thanks a lot guys.

tangor




msg:4400468
 11:41 am on Dec 22, 2011 (gmt 0)

Allow forum/index.html
Disallow forum/

Works for me in real life.

mysite.com/ is not required

Order is important.

Pjman




msg:4400484
 12:34 pm on Dec 22, 2011 (gmt 0)

Thanks tangor. You guys rock.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved