Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- robots.txt for directories with same name


goodroi - 2:21 pm on Aug 5, 2009 (gmt 0)


The search engines will only block the example.com/mypics/ and will crawl example.com/anythinghere/mypics/.

If you want to block all directories that contain the letters "mypics" you need to use wildcards aka pattern matching. This is not officially part of the robots.xt protocol but it is supported by the big three search engines. Be careful some of the smaller crawlers do not support wildcards.


Thread source:: http://www.webmasterworld.com/robots_txt/3965642.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com