I want to disallow languages in my robot.txt

disallow languages in robot.txt

[anthonyinit]

Hello Dear All,

This is my robot.txt

User-agent: *
Disallow: /store/output/
Disallow: /langs/ru.xml
Disallow: /langs/es.xml
Disallow: /langs/de.xml

I want to Disallow all my languages in my website except for English Language.

all my lang are in my langs directory

langs/en.xml
langs/de.xml
langs/ru.xml
langs/es.xml

How can i disallow these languages in my robot.txt

langs/de.xml , langs/ru.xml ,langs/es.xml

Thank you

[not2easy]

You can disallow some robots from crawling the language files, but if you wish to disallow their use on your site, you would need to remove the files.

If you do not want them to be crawled by major robots, use robots.txt - but this does not prevent all robots from reading the files. If the language files exist on your site, only compliant robots honor your robots.txt preferences.

The format would be as you have shared here:

User-agent: *
Disallow: /langs/*.xml

If you do not want your CMS to be able to use other languages, remove the files. This may require further work such as editing a menu where language options are offered. So which are you wanting to do?

BTW, the name of the file that robots request to see if there are files or directories where they are not welcome is "robots.txt" so naming it "robot.txt" might mean they do not read it at all.

[anthonyinit]

Hi @not2easy thank you for ur reply.

User-agent: *
Disallow: /langs/*.xml

if i use the above line it will include my english language as well. "en.xml" i don't want english language to be disallow.

also google index these url of my site which i really don't want.

https://www.example.com/index.php?ccode=RU
https://www.example.com/index.php?ccode=PH
https://www.example.com/index.php?ccode=DE
https://www.example.com/index.php?ccode=ES

[not2easy]

If you want to Disallow crawling of lang files, but allow the /langs/en.xml to be crawled, just add a line to Allow it after the Disallow line. First disallow, then allow:

User-agent: *
Disallow: /langs/*.xml
Allow: /langs/en.xml
Disallow: /*.php?* 

That last line will disallow "anything.php?anything"

You can learn more about the syntax for your robots.txt file from google: [developers.google.com...]

It's always better to be certain and have a reference from the folks whose bots you are trying to talk to.

[anthonyinit]

Great this is what i'm looking for. Thank you so much :)