Robots.txt Guidance

Need to Correct Answer or Solution for My Query

[Irfan Ansari]

I want some folders from website need to block in robots and some not to block in robots.
here the folders path : backoffice/data_content/projects/godrej_anandam_nagpur/Anandam-nagpur1.jpg

If you see here i want to block the back office path or folders in google and want to index my image in google image search so can anyone help me with the solution.

Hope this robots.txt will work
Disallow:*backoffice*
Allow:*data_content*

[keyplyr]

Hello Irfan Ansari and welcome to WebmasterWorld [webmasterworld.com]

The wildcard (*) is only supported by Google. Bing, Yandex, Duck and likely the Asian SEs do not. And you're not using it effectively anyway for what you want.

AFAIK there is no way to allow indexing of a directory beneath a blocked directory using robots text.

One way that may work is to use an htaccess file in the /backoffice/ directory and block with:

X-Robots-Tag: noindex

[developers.google.com...]

But again, while Google supports this method, Bing is intermittent and the other SEs may not at all.

Just a thought... you have a lot of directories nested which is not a good practice for SE indexing. Consider rebuilding your site file heirachy to be more succinct.

[Irfan Ansari]

Sorry brother i still confused not getting you.. can you explain me with the example if you understand my question

[keyplyr]

Disallow:*backoffice* 
Allow:*data_content*

No, your code will not work.

You can try:

User-agent: Googlebot
Disallow: /backoffice/
Allow: /data_content/

User-agent: Googlebot-Image
Disallow: /backoffice/
Allow: /data_content/

[phranque]

At a group-member level, in particular for allow and disallow directives, the most specific rule based on the length of the [path] entry will trump the less specific (shorter) rule. The order of precedence for rules with wildcards is undefined.

https://developers.google.com/search/reference/robots_txt#order-of-precedence-for-group-member-records

i would try:

User-agent: Googlebot
Disallow: /backoffice/
Allow: /backoffice/data_content/

User-agent: Googlebot-Image
Disallow: /backoffice/
Allow: /backoffice/data_content/

[Irfan Ansari]

Thanks So Much keyplyr

[Irfan Ansari]

So Plz guide me which one i need to use and why ....because i don't want to show my backoffice data to google but want my image to get search in google image search as from same backoffice path my images are using.

[keyplyr]

Go with what phranque suggested.

Good luck, it will take several weeks at least to see if it works.

Again, the major obstacle to easily controlling how your site is indexed is your file hierarchy. If you can, restructure your directories so that no file is more than 3 levels from root... 2 is better.

[not2easy]

You can go to your GSC account and test the robots.txt file to see if your robots.txt file performs the way you want it to. Use the URLs you want to test and the robots.txt changes you want to make. Then you can see if it works as you want before you make the change.

[keyplyr]

Ah... that's a great suggestion. I'd forgotten all about that tool.

[Irfan Ansari]

Thanks Guys Will Check and let you know...Once again thanks for your support