Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Block folder from robots.txt file

7:39 am on Jul 29, 2019 (gmt 0)

New User

joined:Dec 14, 2018
posts: 2
votes: 0

I want to block the one folder of the website and want google and other search engines crawler to index the pages inside the folder, i.e.


I don't want www.xyz.com/abc/ to appears in the search engine, so I want to block "abc" folder from robots.txt but I want the crawler to crawl 123.html page (which is inside abc folder).

Can you please guide me on how to write robots.txt code in this situation.

8:42 am on July 29, 2019 (gmt 0)


WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
votes: 235

welcome to WebmasterWorld [webmasterworld.com], naveen10!

you can do this for google which supports the Allow directive in robots.txt - you can see a helpful example here:
Another example file [support.google.com]

not all search engines support the Allow directive.

if you want to prevent that url from being indexed, it would be preferable if you provided a noindex directive in the form of a meta element in the document head or an X-Robots HTTP Response header as described here:
Block search indexing with 'noindex' [support.google.com]

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members