Forum Moderators: goodroi

Message Too Old, No Replies

Block folder from robots.txt file

         

naveen10

7:39 am on Jul 29, 2019 (gmt 0)

5+ Year Member



I want to block the one folder of the website and want google and other search engines crawler to index the pages inside the folder, i.e.

www.xyz.com/abc/123.html

I don't want www.xyz.com/abc/ to appears in the search engine, so I want to block "abc" folder from robots.txt but I want the crawler to crawl 123.html page (which is inside abc folder).

Can you please guide me on how to write robots.txt code in this situation.

Thanks

phranque

8:42 am on Jul 29, 2019 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



welcome to WebmasterWorld [webmasterworld.com], naveen10!

you can do this for google which supports the Allow directive in robots.txt - you can see a helpful example here:
Another example file [support.google.com]

not all search engines support the Allow directive.

if you want to prevent that url from being indexed, it would be preferable if you provided a noindex directive in the form of a meta element in the document head or an X-Robots HTTP Response header as described here:
Block search indexing with 'noindex' [support.google.com]