search engines and IPv6

[lucy24]

As we all know, RIPE IP space is so tight that for the last few years it has been doled out in /22 segments. And that means it's time to start tracking a whole new group of Authorized Ranges.

I noticed it first with Seznam. Almost as soon as my personal site got an IPv6 address (long, irrelevant story) the Seznambot took advantage of this new range: where formerly I saw it from
77.75.76-79
it's now using
2a02:598
almost exclusively. (Weirdly, they still use the IPv4 address for all robots.txt requests.) So far I've consistently met it from 2a02:598:2 and 2a02:598:a, but cursory research reveals that they own the whole /32.

... and that's a whole new way of thinking, in itself. Over in IPv4-land, /32 would be one specific address. You can't get any smaller. In IPv6-land, it's a subsection of a subsection--but you've got another six subsections to go.

So far I haven't met other European search engines--Yandex, Exalead and so on--from IPv6. But it can only be a matter of time. (Hey! Maybe some day Yandex will figure out that if they buy a nice big chunk of IPv6, they can use that for all their crawling.)

[keyplyr]

While the web is definitely moving in the IPv6 direction, and new ranges start out with IPv6 addresses on supported hardware, for reporting purposes it isn't here yet. Why? Because when most of us using shared hosting look at our logs, all we will see is IPv4.

Shared hosting typically uses older servers that 1.) don't support IPv6 reporting and 2.) don't offer that feature (yet) to their hosting customers.

VPS and Dedicated servers can of course config their servers to report IPv6.

However, as older servers are retired and replaced by new machines, and as hosting companies start to add IPv6 reporting, then all ranges will migrate to IPv6 and be read this way.

[lucy24]

Here's another one:

Facebook
2a03:2880::/29

The too-adorable-for-words part is that in the log entries that alerted me to this new IPv6 range, the exact IPs were:
2a03:2880:2130:cff0:face:b00c:0:1
and
2a03:2880:3020:7fe3:face:b00c:0:8000
So far I don't know whether they are intentionally crawling from these specific subsectors, or if it's just a happy coincidence.

[keyplyr]

It's because you have IPv6 turned on that you see IPv6 reported in you server logs. If you did not have IPv6 turned on, you would only see IPv4. If I had my account switched to IPv6 I would likely see several thousand daily requests from IPv6s.

98% of everyone's logs at hosting companies do not see IPv6 addresses reported. IPv4 blocking/routing works with servers using IPv4 even if the request is from an IPv6. This is negotiated.

Note: I'm guesstimating the 98% figure :)

[lucy24]

It's because you have IPv6 turned on that you see IPv6 reported in you server logs.

Er, yes, I know. And your point is ...?

I don't know how long fb has been using the IPv6 range, because they don't often get sent to my personal site (the only one that's IPv6 so far).

[keyplyr]

The point is "Here's another one" could go on forever, but doesn't impact others unless they have switched their server settings to IPv6.

At some point in the future this forum, and the server farm thread, will likely start reporting IPv6 ranges.