Page is a not externally linkable
- Search Engines
-- European Search Engines
---- FYI: New Yandex trick to ignore Robots.txt?


phranque - 11:40 pm on Jul 31, 2012 (gmt 0)


not addressing the why here, but how...

I added a Deny in robots.txt specifically for the Yandex user-agent

are you actually using a "Deny" directive in robots.txt?

if a link has been "published" through the RSS feed, then there is no need to validate them against robots.txt

robots.txt addresses requested resources, not referrers.

yandex has pretty good documentation on their implementation of robots.txt - Using robots.txt - Yandex.Help: webmaster:
http://help.yandex.com/webmaster/?id=1113851 [help.yandex.com]

yandex also has a robots.txt analysis tool - Yandex.Webmaster - Robots.txt analysis:
http://webmaster.yandex.com/robots.xml [webmaster.yandex.com]


Thread source:: http://www.webmasterworld.com/european_search_engines/4432719.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com