Welcome to WebmasterWorld Guest from 54.205.60.171

Forum Moderators: goodroi

Message Too Old, No Replies

Block Rule: Robots.txt - Will Sub Dir also get blocked?

     
7:24 pm on Nov 2, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 19, 2003
posts: 719
votes: 0


A CMS system we use has an enquiry form that when clicked grays out the page...

When scanning the site, it reports it as a 404, and there are 100's..

The url is as follows www.domain.com/product/enquiry_form/product-abc

There are URL's that come after /product/ that I dont want to block. Would I just need to block /enquiry_form/ and not include product?
10:26 pm on Nov 2, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 31, 2005
posts:1108
votes: 0


The first product in the URL a literal or a variable?

Disallow: /product/enquiry_form/
will stop well behaved bots from requesting.
/product/enquiry_form/product-abc
/product/enquiry_form/product-def
etc.

It won't block
/product/somethingother