Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Block Rule: Robots.txt - Will Sub Dir also get blocked?

Dijkgraaf - 10:26 pm on Nov 2, 2010 (gmt 0)


The first product in the URL a literal or a variable?

Disallow: /product/enquiry_form/
will stop well behaved bots from requesting.
/product/enquiry_form/product-abc
/product/enquiry_form/product-def
etc.

It won't block
/product/somethingother

Thread source:: http://www.webmasterworld.com/robots_txt/4225608.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com