homepage Welcome to WebmasterWorld Guest from 54.227.62.141
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Website
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Block Rule: Robots.txt - Will Sub Dir also get blocked?
jaffstar




msg:4225610
 7:24 pm on Nov 2, 2010 (gmt 0)

A CMS system we use has an enquiry form that when clicked grays out the page...

When scanning the site, it reports it as a 404, and there are 100's..

The url is as follows www.domain.com/product/enquiry_form/product-abc

There are URL's that come after /product/ that I dont want to block. Would I just need to block /enquiry_form/ and not include product?

 

Dijkgraaf




msg:4225679
 10:26 pm on Nov 2, 2010 (gmt 0)

The first product in the URL a literal or a variable?

Disallow: /product/enquiry_form/
will stop well behaved bots from requesting.
/product/enquiry_form/product-abc
/product/enquiry_form/product-def
etc.

It won't block
/product/somethingother

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved