If I want to block SE's from a particular directory, can I do this?
Let's say I want www.example.com/widgets directory to be indexed, and also www.examples.com/blue but not www.example.com/blue/widgets, is this exceptable?
User-agent: * Disallow: /blue/widgets/
The reason I ask is I can't find an example that has two directories together, and I don't want to block out 'www.example.com/widgets' or 'www.examples.com/blue' at all, but just the combination.
Will this work? Thanks.
jdMorgan
12:19 am on Mar 1, 2009 (gmt 0)
Robots.txt uses prefix-matching, so your directive will do what you say you want -- Assuming that the rest of the file (if any) is structured properly.
Jim
CWebguy
12:24 am on Mar 1, 2009 (gmt 0)
ok, just making sure because I couldn't find any example that had two directories on one line.
Jonesy
10:08 pm on Mar 1, 2009 (gmt 0)
A followup question:
Would not User-agent: * Disallow: /blue/wi
accomplish the same thing as CWebguy desires -- with the added 'feature' of not revealing the actual sub-directory name(s) to the more scurrilous robots and spiders out there?
g1smd
9:13 am on Mar 16, 2009 (gmt 0)
It would.
With robots.txt you are disallowing all folder names that match the pattern from the left.