homepage Welcome to WebmasterWorld Guest from 54.226.5.29
register, login, search, subscribe, help, library, PubCon, announcements, recent posts, open posts,
Subscribe to WebmasterWorld
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library : Charter : Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Will this work?
Robots.txt
CWebguy




msg:3860104		 12:12 am on Mar 1, 2009 (gmt 0)

If I want to block SE's from a particular directory, can I do this?

Let's say I want www.example.com/widgets directory to be indexed, and also www.examples.com/blue but not www.example.com/blue/widgets, is this exceptable?

User-agent: *
Disallow: /blue/widgets/
The reason I ask is I can't find an example that has two directories together, and I don't want to block out 'www.example.com/widgets' or 'www.examples.com/blue' at all, but just the combination.

Will this work?
Thanks.

 

jdMorgan




msg:3860110		 12:19 am on Mar 1, 2009 (gmt 0)

Robots.txt uses prefix-matching, so your directive will do what you say you want -- Assuming that the rest of the file (if any) is structured properly.

Jim

CWebguy




msg:3860116		 12:24 am on Mar 1, 2009 (gmt 0)

ok, just making sure because I couldn't find any example that had two directories on one line.
Jonesy




msg:3860621		 10:08 pm on Mar 1, 2009 (gmt 0)

A followup question:

Would not

User-agent: *
Disallow: /blue/wi

accomplish the same thing as CWebguy desires -- with the added 'feature' of not revealing the actual sub-directory name(s) to the more scurrilous robots and spiders out there?

g1smd




msg:3871241		 9:13 am on Mar 16, 2009 (gmt 0)

It would.

With robots.txt you are disallowing all folder names that match the pattern from the left.

seosemlive




msg:3873938		 4:25 am on Mar 19, 2009 (gmt 0)

I think it would work
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
WebmasterWorld ® and PubCon ® are a Registered Trademarks of Pubcon Inc.
© Pubcon Inc. 1996-2012 all rights reserved