Forum Moderators: goodroi

Message Too Old, No Replies

I want to disallow languages in my robot.txt

disallow languages in robot.txt

         

anthonyinit

2:08 am on Dec 11, 2020 (gmt 0)

10+ Year Member



Hello Dear All,

This is my robot.txt

User-agent: *
Disallow: /store/output/
Disallow: /langs/ru.xml
Disallow: /langs/es.xml
Disallow: /langs/de.xml


I want to Disallow all my languages in my website except for English Language.

all my lang are in my langs directory

langs/en.xml
langs/de.xml
langs/ru.xml
langs/es.xml


How can i disallow these languages in my robot.txt
langs/de.xml , langs/ru.xml ,langs/es.xml


Thank you

not2easy

2:47 am on Dec 11, 2020 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



You can disallow some robots from crawling the language files, but if you wish to disallow their use on your site, you would need to remove the files.

If you do not want them to be crawled by major robots, use robots.txt - but this does not prevent all robots from reading the files. If the language files exist on your site, only compliant robots honor your robots.txt preferences.

The format would be as you have shared here:
User-agent: *
Disallow: /langs/*.xml

If you do not want your CMS to be able to use other languages, remove the files. This may require further work such as editing a menu where language options are offered. So which are you wanting to do?

BTW, the name of the file that robots request to see if there are files or directories where they are not welcome is "robots.txt" so naming it "robot.txt" might mean they do not read it at all.

anthonyinit

3:23 am on Dec 11, 2020 (gmt 0)

10+ Year Member



Hi @not2easy thank you for ur reply.
User-agent: *
Disallow: /langs/*.xml

if i use the above line it will include my english language as well. "en.xml" i don't want english language to be disallow.

also google index these url of my site which i really don't want.
https://www.example.com/index.php?ccode=RU
https://www.example.com/index.php?ccode=PH
https://www.example.com/index.php?ccode=DE
https://www.example.com/index.php?ccode=ES

not2easy

3:51 am on Dec 11, 2020 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



If you want to Disallow crawling of lang files, but allow the /langs/en.xml to be crawled, just add a line to Allow it after the Disallow line. First disallow, then allow:
User-agent: *
Disallow: /langs/*.xml
Allow: /langs/en.xml
Disallow: /*.php?*

That last line will disallow "anything.php?anything"

You can learn more about the syntax for your robots.txt file from google: [developers.google.com...]

It's always better to be certain and have a reference from the folks whose bots you are trying to talk to.

anthonyinit

4:12 am on Dec 11, 2020 (gmt 0)

10+ Year Member



Great this is what i'm looking for. Thank you so much :)