Welcome to WebmasterWorld Guest from 3.227.249.234

Forum Moderators: goodroi

Block Only Yandex Bot

Know the command of how block only yandex bot on robots,txt file

     
5:07 am on Aug 27, 2019 (gmt 0)

New User from IN 

joined:July 25, 2019
posts: 2
votes: 0


Will be used only by the main indexing robot

User-agent: YandexBot
Disallow: /


Will be used by all Yandex robots

User-agent: Yandex
Disallow: /
5:15 am on Aug 27, 2019 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:10457
votes: 1091


@nehasurya ... Welcome to Webmasterworld.

You might look at yandex's own directives page:

[yandex.com...]
4:25 pm on Aug 27, 2019 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15872
votes: 869


Obligatory quibble: Nobody is “blocked” in robots.txt. They are “denied” or “disallowed”. It’s the difference between installing a deadbolt and putting up a “No Admittance” sign. With compliant robots it should make no difference--yandex is currently compliant--but it’s important to keep a firm grip on the difference.
10:58 pm on Aug 27, 2019 (gmt 0)

Preferred Member from CA 

Top Contributors Of The Month

joined:Feb 7, 2017
posts:575
votes: 59


robots.txt is a guideline that politely asks bots to not visit. The good ones, Yandex is a good one, should follow the guideline. Most other bots completely ignore robots.txt. In fact most bots don't even read the robots.txt. For bots other than the good ones, expect the robots.txt to be completely ineffective.
5:15 am on Aug 29, 2019 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:10457
votes: 1091


I use robots.txt to expose the bad actors. Those that don't honor it are fed 403s until I can determine any benefit of putting them in my robots.txt "allow" section.

YMMV

.