Welcome to WebmasterWorld Guest from 54.198.185.204

Forum Moderators: Ocean10000 & incrediBILL & keyplyr

Message Too Old, No Replies

Yandex gets fatter

     
2:05 am on Nov 11, 2012 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:13061
votes: 303


37.140.128.0/18
Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)

Like the other yandexbots I've met, they pick an IP and stick with it. Here it was 37.140.141.13.

I've yet to see them on my own site, but a rare visit to art studio's logs

:: insert boilerplate about finding something while looking for something else (in this case iPad misbehavior, see nearby post) ::

found them all over the place, alternating with the familiar 199.21.64.0/20 and a handful of 95.108.128.0/17.

They didn't go anywhere they weren't supposed to.
9:54 pm on Nov 11, 2012 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3101
votes: 3


Thanks for the new range, Lucy. Just ran a DNS check on the full range and the word "spider" occurs only in the range 37.140.141.1 - 37.140.141.36

Of these, 37.140.141.1 - 37.140.141.12 are basic "spider"; the others are "image-spider".
10:11 pm on Nov 11, 2012 (gmt 0)

Preferred Member

10+ Year Member

joined:May 8, 2005
posts:408
votes: 0


Yandex has a nice index, it kinda reminds me of a search engine. I have seen a few referrals and noticed that dogpile is tossing them into their meta results. They do crawl a lot but I'll cut them slack.
10:49 pm on Nov 11, 2012 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:6152
votes: 75


I get a fair amount of traffic from Yandex. Their bot behaves and I have never had a problem.

Yandex has a nice index, it kinda reminds me of a search engine.

Probably because they are.
5:04 am on Jan 31, 2013 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:13061
votes: 303


:: bump ::

Here's another one:

37.9.64.0/18
possibly constrained to
37.9.84.0/22 (37.9.84-87)

Only met them twice so far.
exact IP: 37.9.84.253
full UA: Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/ bots)

Along with the favicon (first visit only), they picked up the front page both times.

No relation to 37.9.0.0/18 (main activity in the 37.9.50's, I think) which has been mentioned in a few threads hereabouts. But while cross-checking the UA I found this Yandex page [help.yandex.com] (in English) which I don't remember seeing before.

Yandex has a nice index, it kinda reminds me of a search engine.
Probably because they are.

I thought he was being satirical ;)
8:20 am on Jan 31, 2013 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:6152
votes: 75



There are many IP addresses that Yandex robots can originate from, and these IP addresses are subject to change. We are therefore unable to offer a list of IP addresses and we do not recommend using a filter based on IP addresses.

This is interesting.
8:34 am on Jan 31, 2013 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:13061
votes: 303


At least so far the IPs do seem to belong to Yandex. It isn't as hopeless as, say, the MJ12bot which can come from absolutely anywhere-- including a good many blocked server farms. And, conversely, you don't see a lot of Yandexbot spoofers. Fake yandsearch* in referers, yes. Fake UAs, not so much.


* I did some quickie experimenting. The fake referer isn't any different from a real one. Except that currently mine all end in &lr=213. I don't know what "lr" is, only that it doesn't correspond to "cd".
11:57 am on Jan 31, 2013 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 13, 2005
posts: 128
votes: 0


dstiles: "Just ran a DNS check on the full range and the word "spider" occurs only in the range 37.140.141.1 - 37.140.141.36"

Could you share how you did that? I've asked that question here before and never really gotten a good response.
10:31 pm on Jan 31, 2013 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3101
votes: 3


I use dig on linux, run in a bash shell script. I run the checks against one of the relevant DNS servers (eg yandex for yandex, ms for bingbot etc).

Slow but a) I'm in no hurry for the results and b) I worry that a fast DNS scan would get me blocked.

Not sure how to run this under windows but there is probably an equivalent to dig.
11:14 pm on Jan 31, 2013 (gmt 0)

Senior Member from FR 

WebmasterWorld Senior Member leosghost is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Feb 15, 2004
posts:6717
votes: 230


Not sure how to run this under windows but there is probably an equivalent to dig.


I was going to suggest "sam spade" ..(was a "multi tools" for win ) ..and then went for a look ..and "sam" is gone :( .."where have all the..."

So ..going to pour myself a "leapfrog"..and make sure there are no kids on my lawn..
8:30 pm on Feb 1, 2013 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3101
votes: 3


I used to use sam when I was using windows machines for everyday activities but gave up when various bits of it stopped working. At the time there was still a web version running but I haven't looked for several years. It's possible that NS-Batch might still be available and capable of doing this but I've only used that once (recently) for tracking a list of IPs and can't recall its full facilities.
9:43 pm on Feb 1, 2013 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:13061
votes: 303


At the time there was still a web version running

Heh. I didn't know there was anything but a www version. It must have closed up shop eons ago; I even took it off my bookmarks. Currently dot com times out and dot org gives you the "It Works!" header which I think is generated by testing something-or-other but, er, I can't remember what. Cursory whois'ing says that dot org is the one we want :(.
9:57 pm on Feb 1, 2013 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member

joined:June 14, 2010
posts:985
votes: 0


...the "It Works!" header...

It's the default message that an Apache server homepage displays once it's set up properly.