Welcome to WebmasterWorld Guest from 18.207.132.114

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

New Googlebot locations in South Asia and Hong Kong

     
4:05 am on Nov 2, 2017 (gmt 0)

Junior Member

joined:Aug 22, 2017
posts: 71
votes: 3


I've noticed that my India-focused website, which used to be crawled from Mountain View, California when it used to be on Cloudflare, is being crawled from another location close by (within Asia) now that it's no longer on Cloudflare. There seems to be no crawling operations in India, but there seems to be crawlers originating in KL, HK and Singapore (possibly more) according to my testing.
Similarly, in Europe, there's one in London and another in Paris.
Question is -- does this mean that one should try to host the website close to the target location? In other words, if my website is hosted in India (and hence indexed using the India-specific Googlebot), might it rank higher for Indian searches?
7:40 am on Nov 2, 2017 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 893


Have you verified these ranges as being official Googlebot crawl ranges?

I've seen no announcement regarding new Googlebot crawl ranges. This is likely an imposter. Googlebot is the most faked UA on the internet.

Please post these new ranges. Thanks.
8:08 am on Nov 2, 2017 (gmt 0)

Junior Member

joined:Aug 22, 2017
posts: 71
votes: 3


hostname lookup and then whois gives this --

Domain Name: 1E100.NET
Registry Domain ID: 1570253561_DOMAIN_NET-VRSN
Registrar WHOIS Server: whois.markmonitor.com
Registrar URL: http://www.markmonitor.com
Updated Date: 2010-09-15T17:45:17Z
Creation Date: 2009-09-25T05:40:03Z
Registry Expiry Date: 2019-09-25T05:40:03Z
Registrar: MarkMonitor Inc.
Registrar IANA ID: 292
Registrar Abuse Contact Email: abusecomplaints@markmonitor.com
Registrar Abuse Contact Phone: +1.2083895740
Domain Status: clientDeleteProhibited [icann.org...]
Domain Status: clientRenewProhibited [icann.org...]
Domain Status: clientTransferProhibited [icann.org...]
Domain Status: clientUpdateProhibited [icann.org...]
Domain Status: serverDeleteProhibited [icann.org...]
Domain Status: serverTransferProhibited [icann.org...]
Domain Status: serverUpdateProhibited [icann.org...]
Name Server: NS1.GOOGLE.COM
Name Server: NS2.GOOGLE.COM
Name Server: NS3.GOOGLE.COM
Name Server: NS4.GOOGLE.COM
DNSSEC: unsigned

[edited by: Robert_Charlton at 9:13 am (utc) on Nov 2, 2017]
[edit reason] delinked promotional link for registrar [/edit]

8:22 am on Nov 2, 2017 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 893


Where are the IP ranges?

Just a FYI - Google owns a lot of IP ranges. Only a small subset are dedicated as Googlebot crawl ranges.

Google leases thousands of other IPs to anyone who wants to use them. When you look them up they may say they belong to Google, but they may not be official Googlebot crawl ranges.

Many questionable agents will lease one of those other Google ranges and fake the user agent to appear they are Googlebot. They do this to get access to web properties that otherwise may be blocked.
10:13 am on Nov 2, 2017 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator robert_charlton is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2000
posts:12411
votes: 416


elos42, thanks for your interest in completeness, though I'm not sure that all the definitions of ICANN domain status codes (clientDeleteProhibited, clientTransferProhibited, etc etc) you linked to are likely to be of much to do with your question.

As keyplyr suggests, and he's the moderator for the Search Engine Spider and User Agent Identification forum on WebmasterWorld, what's likely to be more of interest to webmasters are the IP ranges of these. Perhaps you and he can discuss in the thread how best to identify these and their association with Google, as that might well be of interest to some members.

I've disabled your direct link to marketmonitor.com, as it's essentially a promotional link for the company. The Google search answer box has this to say about who marketmonitor is...
MarkMonitor is an American company which offers brand protection service against fraud and cybersquatting, it is the ICANN accredited domain registrar for the largest internet companies including Facebook, Google, Microsoft, Apple and other major companies around the world.

Regarding the domain name you've identified... "Domain Name: 1E100.NET", a Google help article I found has this description...
What is 1e100.net?
[support.google.com...]

1e100.net is a Google-owned domain name used to identify the servers in our network.

Following standard industry practice, we make sure each IP address has a corresponding hostname. In October 2009, we started using a single domain name to identify our servers across all Google products, rather than use different product domains such as youtube.com, blogger.com, and google.com. We did this for two reasons: first, to keep things simpler, and second, to proactively improve security by protecting against potential threats such as cross-site scripting attacks.

Most typical Internet users will never see 1e100.net, but we picked a Googley name for it just in case (1e100 is scientific notation for 1 googol).


Regarding the effect of hosting location, see this discussion, one of several I believe we've had recently on the topic...

Does Web Hosting Server Location Change affect SEO?
Jul 31, 2017
https://www.webmasterworld.com/google/4860726.htm [webmasterworld.com]

I don't think that proximity to the source of these spiders will have much if anything to do with your ranking in Indian searches. Read the thread.

11:42 am on Nov 2, 2017 (gmt 0)

Junior Member

joined:Aug 22, 2017
posts: 71
votes: 3


thanks much. The IP in question is 172.217.31.100. That's for India. I've shut down most of the other servers in other countries, and therefore do not have those IPs at hand.
4:32 pm on Nov 2, 2017 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15941
votes: 889


A few years back they were talking about Locale-Aware Crawl Configurations [webmasterworld.com] but I don't think anyone has seen any concrete evidence that they are actually doing it.
5:53 pm on Nov 2, 2017 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 893


@elos42, thanks for the IP address.

172.217.31.100 is currently assigned to: kul08s08-in-f4.1e100.net

This company leases this range from:
Google
172.217.0.0 - 172.217.255.255
172.217.0.0/16

This is *not* a designated crawl range for Googlebot.

The article Lucy24 referenced: [support.google.com...] does say:
In order to address crawling and indexing of locale-adaptive content, we use locale-aware crawling to better surface your content for searchers around the world... Googlebot can now use IP addresses that appear to come from other countries
but Googlebot would *seemingly* still come from a designated crawl range. However, I'm not sure about this, so without more explicit documentation, I can't say whether the above mentioned range is Googlebot or not.

Regarding your OP question, it is always a good idea to host your site at a server based near your target audience for speed, but it is more of a content/audience issue. So yes, if your target audience is mostly from India, it is better (but not essential) to use a hosting company in India... but there in no indication that Google will boost your ranking for doing this.
10:10 pm on Nov 2, 2017 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Sept 25, 2005
posts:2091
votes: 370


Note that the Google Cloud platform now has a region in Mumbai, India, where anyone can spin up a server.
5:09 am on Nov 11, 2017 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 893


How to verify the *real* Googlebot:
[webmasters.googleblog.com...]

Other Google agents:
[support.google.com...]
5:24 am on Nov 11, 2017 (gmt 0)

Junior Member

joined:Aug 22, 2017
posts: 71
votes: 3


it is always a good idea to host your site at a server based near your target audience for speed, but it is more of a content/audience issue. So yes, if your target audience is mostly from India, it is better (but not essential) to use a hosting company in India.

Ok. It's just that for the same amount of money, I can get a server that's about twice as fast (CPU and RAM wise) in Singapore (plus free backup) compared to India. The difference in latency is about 40 ms, depending on where the user is.

Bandwidth costs and taxes are higher in India. Besides, I do get about 10-15% traffic (that account for about 35% of my revenue) from outside India, and these customers would definitely be better served from Singapore I think.

The quality of network provided by my Singapore provider is superior in many ways -- for example, I get a 500 Mbps port that has always tested at 500 Mbps. In India, it's very difficult to get any with more than 100 Mbps. (Yes, I do hit the 100 Mbps mark sometimes -- twice or thrice a month for a few minutes each.)

From an India user exp point of view, the difference in terms of load time for cached pages (90% of the traffic) is one between 'blazing fast' and 'pretty fast'.

For now, I'm going with 'pretty fast' from Singapore. But I could make the switch (and upgrade my hosting budget) if this also has search ranking implications.

Anyway, thank you for your advice. I'll see if there's any more information on this.

PS: I also discovered that if I host my website in India for a sufficient length of time, I do indeed seem to get crawled from an Indian IP address (as in, there's a sudden drop in 'time taken to download an average page' in GSC data.)