New Googlebot locations in South Asia and Hong Kong

[elos42]

I've noticed that my India-focused website, which used to be crawled from Mountain View, California when it used to be on Cloudflare, is being crawled from another location close by (within Asia) now that it's no longer on Cloudflare. There seems to be no crawling operations in India, but there seems to be crawlers originating in KL, HK and Singapore (possibly more) according to my testing.
Similarly, in Europe, there's one in London and another in Paris.
Question is -- does this mean that one should try to host the website close to the target location? In other words, if my website is hosted in India (and hence indexed using the India-specific Googlebot), might it rank higher for Indian searches?

[keyplyr]

Have you verified these ranges as being official Googlebot crawl ranges?

I've seen no announcement regarding new Googlebot crawl ranges. This is likely an imposter. Googlebot is the most faked UA on the internet.

Please post these new ranges. Thanks.

[elos42]

hostname lookup and then whois gives this --

Domain Name: 1E100.NET
Registry Domain ID: 1570253561_DOMAIN_NET-VRSN
Registrar WHOIS Server: whois.markmonitor.com
Registrar URL: http://www.markmonitor.com
Updated Date: 2010-09-15T17:45:17Z
Creation Date: 2009-09-25T05:40:03Z
Registry Expiry Date: 2019-09-25T05:40:03Z
Registrar: MarkMonitor Inc.
Registrar IANA ID: 292
Registrar Abuse Contact Email: [e-mail]
Registrar Abuse Contact Phone: [phone]
Domain Status: clientDeleteProhibited [icann.org...]
Domain Status: clientRenewProhibited [icann.org...]
Domain Status: clientTransferProhibited [icann.org...]
Domain Status: clientUpdateProhibited [icann.org...]
Domain Status: serverDeleteProhibited [icann.org...]
Domain Status: serverTransferProhibited [icann.org...]
Domain Status: serverUpdateProhibited [icann.org...]
Name Server: NS1.GOOGLE.COM
Name Server: NS2.GOOGLE.COM
Name Server: NS3.GOOGLE.COM
Name Server: NS4.GOOGLE.COM
DNSSEC: unsigned

[edited by: Robert_Charlton at 9:13 am (utc) on Nov 2, 2017]
[edit reason] delinked promotional link for registrar [/edit]

[keyplyr]

Where are the IP ranges?

Just a FYI - Google owns a lot of IP ranges. Only a small subset are dedicated as Googlebot crawl ranges.

Google leases thousands of other IPs to anyone who wants to use them. When you look them up they may say they belong to Google, but they may not be official Googlebot crawl ranges.

Many questionable agents will lease one of those other Google ranges and fake the user agent to appear they are Googlebot. They do this to get access to web properties that otherwise may be blocked.

[Robert Charlton]

elos42, thanks for your interest in completeness, though I'm not sure that all the definitions of ICANN domain status codes (clientDeleteProhibited, clientTransferProhibited, etc etc) you linked to are likely to be of much to do with your question.

As keyplyr suggests, and he's the moderator for the Search Engine Spider and User Agent Identification forum on WebmasterWorld, what's likely to be more of interest to webmasters are the IP ranges of these. Perhaps you and he can discuss in the thread how best to identify these and their association with Google, as that might well be of interest to some members.

I've disabled your direct link to marketmonitor.com, as it's essentially a promotional link for the company. The Google search answer box has this to say about who marketmonitor is...

MarkMonitor is an American company which offers brand protection service against fraud and cybersquatting, it is the ICANN accredited domain registrar for the largest internet companies including Facebook, Google, Microsoft, Apple and other major companies around the world.

Regarding the domain name you've identified... "Domain Name: 1E100.NET", a Google help article I found has this description...

What is 1e100.net?
[support.google.com...]

1e100.net is a Google-owned domain name used to identify the servers in our network.

Following standard industry practice, we make sure each IP address has a corresponding hostname. In October 2009, we started using a single domain name to identify our servers across all Google products, rather than use different product domains such as youtube.com, blogger.com, and google.com. We did this for two reasons: first, to keep things simpler, and second, to proactively improve security by protecting against potential threats such as cross-site scripting attacks.

Most typical Internet users will never see 1e100.net, but we picked a Googley name for it just in case (1e100 is scientific notation for 1 googol).

Regarding the effect of hosting location, see this discussion, one of several I believe we've had recently on the topic...

Does Web Hosting Server Location Change affect SEO?
Jul 31, 2017
https://www.webmasterworld.com/google/4860726.htm [webmasterworld.com]

I don't think that proximity to the source of these spiders will have much if anything to do with your ranking in Indian searches. Read the thread.

[elos42]

thanks much. The IP in question is 172.217.31.100. That's for India. I've shut down most of the other servers in other countries, and therefore do not have those IPs at hand.

[lucy24]

A few years back they were talking about Locale-Aware Crawl Configurations [webmasterworld.com] but I don't think anyone has seen any concrete evidence that they are actually doing it.

[keyplyr]

@elos42, thanks for the IP address.

172.217.31.100 is currently assigned to: kul08s08-in-f4.1e100.net

This company leases this range from:
Google
172.217.0.0 - 172.217.255.255
172.217.0.0/16

This is *not* a designated crawl range for Googlebot.

The article Lucy24 referenced: [support.google.com...] does say:

In order to address crawling and indexing of locale-adaptive content, we use locale-aware crawling to better surface your content for searchers around the world... Googlebot can now use IP addresses that appear to come from other countries

but Googlebot would *seemingly* still come from a designated crawl range. However, I'm not sure about this, so without more explicit documentation, I can't say whether the above mentioned range is Googlebot or not.

Regarding your OP question, it is always a good idea to host your site at a server based near your target audience for speed, but it is more of a content/audience issue. So yes, if your target audience is mostly from India, it is better (but not essential) to use a hosting company in India... but there in no indication that Google will boost your ranking for doing this.

[robzilla]

Note that the Google Cloud platform now has a region in Mumbai, India, where anyone can spin up a server.

[keyplyr]

How to verify the *real* Googlebot:
[webmasters.googleblog.com...]

Other Google agents:
[support.google.com...]

[elos42]

it is always a good idea to host your site at a server based near your target audience for speed, but it is more of a content/audience issue. So yes, if your target audience is mostly from India, it is better (but not essential) to use a hosting company in India.

Ok. It's just that for the same amount of money, I can get a server that's about twice as fast (CPU and RAM wise) in Singapore (plus free backup) compared to India. The difference in latency is about 40 ms, depending on where the user is.

Bandwidth costs and taxes are higher in India. Besides, I do get about 10-15% traffic (that account for about 35% of my revenue) from outside India, and these customers would definitely be better served from Singapore I think.

The quality of network provided by my Singapore provider is superior in many ways -- for example, I get a 500 Mbps port that has always tested at 500 Mbps. In India, it's very difficult to get any with more than 100 Mbps. (Yes, I do hit the 100 Mbps mark sometimes -- twice or thrice a month for a few minutes each.)

From an India user exp point of view, the difference in terms of load time for cached pages (90% of the traffic) is one between 'blazing fast' and 'pretty fast'.

For now, I'm going with 'pretty fast' from Singapore. But I could make the switch (and upgrade my hosting budget) if this also has search ranking implications.

Anyway, thank you for your advice. I'll see if there's any more information on this.

PS: I also discovered that if I host my website in India for a sufficient length of time, I do indeed seem to get crawled from an Indian IP address (as in, there's a sudden drop in 'time taken to download an average page' in GSC data.)