Page is a not externally linkable
phranque - 11:40 pm on Jul 31, 2012 (gmt 0)
not addressing the why here, but how...
I added a Deny in robots.txt specifically for the Yandex user-agent
are you actually using a "Deny" directive in robots.txt?
if a link has been "published" through the RSS feed, then there is no need to validate them against robots.txt
robots.txt addresses requested resources, not referrers.
yandex has pretty good documentation on their implementation of robots.txt - Using robots.txt - Yandex.Help: webmaster:
http://help.yandex.com/webmaster/?id=1113851 [help.yandex.com]
yandex also has a robots.txt analysis tool - Yandex.Webmaster - Robots.txt analysis:
http://webmaster.yandex.com/robots.xml [webmaster.yandex.com]