Obligatory quibble: Nobody is “blocked” in robots.txt. They are “denied” or “disallowed”. It’s the difference between installing a deadbolt and putting up a “No Admittance” sign. With compliant robots it should make no difference--yandex is currently compliant--but it’s important to keep a firm grip on the difference.
TorontoBoy
10:58 pm on Aug 27, 2019 (gmt 0)
robots.txt is a guideline that politely asks bots to not visit. The good ones, Yandex is a good one, should follow the guideline. Most other bots completely ignore robots.txt. In fact most bots don't even read the robots.txt. For bots other than the good ones, expect the robots.txt to be completely ineffective.
tangor
5:15 am on Aug 29, 2019 (gmt 0)
I use robots.txt to expose the bad actors. Those that don't honor it are fed 403s until I can determine any benefit of putting them in my robots.txt "allow" section.
