Welcome to WebmasterWorld Guest from 3.227.233.55

Forum Moderators: goodroi

Block folder from robots.txt file

     
7:39 am on Jul 29, 2019 (gmt 0)

New User

joined:Dec 14, 2018
posts: 2
votes: 0


I want to block the one folder of the website and want google and other search engines crawler to index the pages inside the folder, i.e.

www.xyz.com/abc/123.html

I don't want www.xyz.com/abc/ to appears in the search engine, so I want to block "abc" folder from robots.txt but I want the crawler to crawl 123.html page (which is inside abc folder).

Can you please guide me on how to write robots.txt code in this situation.

Thanks
8:42 am on July 29, 2019 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:11817
votes: 235


welcome to WebmasterWorld [webmasterworld.com], naveen10!

you can do this for google which supports the Allow directive in robots.txt - you can see a helpful example here:
Another example file [support.google.com]

not all search engines support the Allow directive.

if you want to prevent that url from being indexed, it would be preferable if you provided a noindex directive in the form of a meta element in the document head or an X-Robots HTTP Response header as described here:
Block search indexing with 'noindex' [support.google.com]
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members