Something new from Yandex?

VisualParser

[GaryK]

Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9) Gecko VisualParser/3.0
93.158.136.nnn
samui.yandex.net

I couldn't find anything about this on their site or in Google.

It did not read robots.txt and it did take disallowed pages.

This surprised me because Yandex is one of the few search engines I allow to crawl my sites because I get a lot of traffic from them. In the past they have always read and respected robots.txt.

[GaryK]

More new stuff from Yandex. This one was more in line with what I expect from Yandex. It read and respected robots.txt. It took pages at an acceptable rate.

YaDirectBot/1.0
77.88.57.nnn
nastenka02d.yandex.ru

[edited by: incrediBILL at 6:41 pm (utc) on June 9, 2009]
[edit reason] Obscured IPs [/edit]

[enigma1]

Here is something else from yandex

93.158.151.nn - - [08/Jun/2009:23:46:31 -0400] "GET / HTTP/1.1" 301 5 "-" "YandexSomething/1.0"

yandexsomething as the ua?

[dstiles]

I had that yesterday, as well. I only found one report in English about it, from a bots listing site that said it had been around for a few months. I've had yandex allowed for some time as being a reasonable bot but that UA is a bit of a puzzler. :)

[enigma1]

When you say a reasonable bot what do you mean? If you want to see here are couple of other entries from yandex.

93.158.136.nnn - - [13/Apr/2009:11:00:37 -0400] "GET / HTTP/1.1" 301 5 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

93.158.156.nnn - - [27/Feb/2009:05:44:03 -0500] "GET / HTTP/1.1" 301 5 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts)"

Anyways for me the UA alone is not a factor to block access to a spider. However the target audience is and unless is international with english as the primary language it gets blocked.

[Hobbs]

93.158.150.nn YandexSomething/1.0
213.180.207.nnn Yandex/1.02.000 (compatible; Win16; F)

It was taking 22 pages per minute
Best effort failed to find a robots page on their site
Got itself blocked yesterday, 'big search' engine in Russia or not, life's too short.

[dstiles]

Hmm. I've been allowing it because some of my customers trade internationally, although they seem to communicate in English. Maybe it's time I went through my SE logs.

[GaryK]

Thought I had reported on this some months ago. Sorry about that. When I first saw this the only files it took were RSS/XML files. I put it in my feed syndicators category. It visits regularly and that's all it's ever taken from the one site it visits.