homepage Welcome to WebmasterWorld Guest from 54.163.72.86
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Help with Disallow
Not sure if I got this right...
BillyS




msg:1529306
 3:56 am on Nov 14, 2005 (gmt 0)

I just consolidated several sections and I don't want crawlers to index those pages (removed) anymore.

The old URL was of the form:

www.sitename.com/section/7/14/
www.sitename.com/section/7/21/

If I put in this:

Disallow: /7/

That should stop indexing of these pages (right?)

 

MarkHutch




msg:1529307
 4:16 am on Nov 14, 2005 (gmt 0)

From your example your robots.txt file should say..

Disallow: /section/7/

If the section is a real subdirectory. If it's not t hen you are correct.

Mark

Dijkgraaf




msg:1529308
 8:52 pm on Nov 14, 2005 (gmt 0)

MarkHutch is correct, but you would be even better of having
Disallow: /section/7
without the traling slash, this is just in case there was a link pointing to that directory that served the default page. That of course is as long as there isn't another directory e.g /section/71 that you still want indexed :-)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved