homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Will this work?

 12:12 am on Mar 1, 2009 (gmt 0)

If I want to block SE's from a particular directory, can I do this?

Let's say I want www.example.com/widgets directory to be indexed, and also www.examples.com/blue but not www.example.com/blue/widgets, is this exceptable?

User-agent: *
Disallow: /blue/widgets/

The reason I ask is I can't find an example that has two directories together, and I don't want to block out 'www.example.com/widgets' or 'www.examples.com/blue' at all, but just the combination.

Will this work?



 12:19 am on Mar 1, 2009 (gmt 0)

Robots.txt uses prefix-matching, so your directive will do what you say you want -- Assuming that the rest of the file (if any) is structured properly.



 12:24 am on Mar 1, 2009 (gmt 0)

ok, just making sure because I couldn't find any example that had two directories on one line.


 10:08 pm on Mar 1, 2009 (gmt 0)

A followup question:

Would not

User-agent: *
Disallow: /blue/wi

accomplish the same thing as CWebguy desires -- with the added 'feature' of not revealing the actual sub-directory name(s) to the more scurrilous robots and spiders out there?


 9:13 am on Mar 16, 2009 (gmt 0)

It would.

With robots.txt you are disallowing all folder names that match the pattern from the left.


 4:25 am on Mar 19, 2009 (gmt 0)

I think it would work

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved