Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- robots.txt disallow searching reoccuring folders

g1smd - 8:30 pm on Aug 29, 2011 (gmt 0)


The pattern matching is "from the left".

So /images disallows any URL path "beginning" /images

And /*images disallows any URL path "containing" images

Thread source:: http://www.webmasterworld.com/robots_txt/4356400.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com