Googlebot and iPhone

[wilderness]

There have been multiple recent threads on Google deviating from its former standard practices.

A Google Boy using an iPhone, may in fact be a valid bot, however I'm not interested.

(It's worth noting that iPhone's make regular requests for single pages absent any supporting files (I've been fairly tolerant/lax of these requests))

Google & IPhone needs to get their prospective acts in order.

RewriteCond %{HTTP_USER_AGENT} iPhone
RewriteCond %{REMOTE_ADDR} ^^66\.249\.(6[4-9]|[7][0-9])\.
RewriteRule .* - [F]

NONE of these page requests included supporting files.

66.249.79.59 - - [04/Aug/2014:20:24:47 -0600] "GET /MySub/MyPage.html HTTP/1.1" 200 27040 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.59 - - [04/Aug/2014:20:36:34 -0600] "GET /DifferentFolder/DifferentSubFolder/DifferentPage.html.html HTTP/1.1" 200 16412 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.72 - - [04/Aug/2014:21:17:25 -0600] "GET /SameFolder/SameSub/DifferentPage.html HTTP/1.1" 200 10366 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.59 - - [04/Aug/2014:21:24:28 -0600] "GET /SameFolder/SameSub/DifferentPage.html HTTP/1.1" 200 17851 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.72 - - [04/Aug/2014:21:31:02 -0600] "GET /SameFolder/DifferentSubFolder/DifferentPage.html HTTP/1.1" 304 162 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.59 - - [05/Aug/2014:00:02:56 -0600] "GET /SameFolder/DifferentPage.html HTTP/1.1" 304 162 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.46 - - [05/Aug/2014:05:01:03 -0600] "GET /DifferentFolder/DifferentPage.htm HTTP/1.1" 304 162 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.59 - - [05/Aug/2014:05:06:32 -0600] "GET /SameFolder/DifferentSubFolder/DifferentPage.html HTTP/1.1" 200 5494 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

[keyplyr]

No problem, this is Googlebot testing to see if your site is responsive to mobile devices (various screen resolutions.)

There is growing speculation that if your site is not optimized for mobile screens, it will not do well in the mobile SERP in the very near future (if not already.)

Desk-top is loosing market share and mobile (phones/tablets) is taking over very fast. Site's that don't keep up may become archaic. I'm actually right in the middle of coding my sites to be responsive. Should be done in a few weeks.

[Samizdata]

I'm not interested

Your prerogative, Don, as always.

If your site does not offer different content to smartphones then you may not need the extra crawls.

Update October 2013: The new Googlebot-Mobile for smartphones user-agent is: Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

The content crawled by smartphone Googlebot-Mobile will be used primarily to improve the user experience on mobile search. For example, the new crawler may discover content specifically optimized to be browsed on smartphones as well as smartphone-specific redirects.

One new feature we’re also launching that uses these signals is Skip Redirect for Smartphone-Optimized Pages. When we discover a URL in our search results that redirects smartphone users to another URL serving smartphone-optimized content, we change the link target shown in the search results to point directly to the final destination URL. This removes the extra latency the redirect introduces leading to a saving of 0.5-1 seconds on average when visiting landing page for such search results. Update 9 August 2013: If a site uses separate URLs to serve desktop and smartphone users, and if we discover accurate rel-alternate-media annotations linking the desktop and smartphone pages, our algorithms may change the link target shown in the search results to point directly to the smartphone page.

Source: [googlewebmastercentral.blogspot.co.uk...]

...

[lucy24]

Googlebot/2.1; +http://www.google.com/bot.html

In other words, the UA name element -Mobile is conspicious by its absence?

I haven't seen this yet. They must like you better than me. Got a nasty feeling this is not the first time I have made this observation.

[keyplyr]

AFAIK Googlebot dropped "Mobile" form it's mobile checker several weeks ago, at least I haven't seen it for that long. What I see are variations of the UA Don posted but sometimes instead of iPhone the type is DocoMo or Linux or Android, etc.

[Samizdata]

In other words, the UA name element -Mobile is conspicious by its absence?

Looks like they simply updated the UA again: [support.google.com...]

This particular iteration seems to be officially named "Google Smartphone".

The only remarkable aspect is that the UA uses "iPhone" rather than "Android".

...

[wilderness]

There is growing speculation that if your site is not optimized for mobile screens, it will not do well in the mobile SERP in the very near future (if not already.)

If your site does not offer different content to smartphones then you may not need the extra crawls.

Many thanks to all for the feedback.

FWIW, my pages were never intended for hand held devices (phone or otherwise). It's my plan (with recent delays added due to relocation priorities) to vacate both the internet and widgets in less than three years, and during that interim, I've no plans to expand my horizons for mobile devices.

Let 'em go somewhere else and get the widget data that I've available (good luck with that).

[wilderness]

This thing has gained a healthier appetite since eating 403's.

Their primary bot is still crawling simultaneously.

[keyplyr]

This thing has gained a healthier appetite since eating 403's.

Yup, do some reading over in the Mobile forum. Looks like Google is checking all sites and is now demoting (or purging) those that aren't mobile responsive from their mobile index.

My main site, built with expandable containers, actually looked OK on most devices and used to get a pretty good rating on the mobile scorecard, but taken a hit last few months probably due to Google's big move toward determining *true* mobile responsiveness.

[wilderness]

demoting (or purging) those that aren't mobile responsive from their mobile index.

That'll make life a bit easier during the interim of my exodus ;)

[not2easy]

I just went over and took a look because I do recall reading that Google had changed their Mobile UA - and at [support.google.com...] Google's UA Page I see that they use "two" but one of those two has varieties.

Googlebot-Mobile UA is:
[various mobile device types] (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

The Smartphone UA is:
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

The "[various mobile device types]" in the Googlebot-Mobile UA is where you see Samsung, DoCoMo, et al.

What else I found out is that they no longer will disclose what IPs they crawl from, to prevent whitelisting. :(

[keyplyr]

It's my plan... to vacate both the internet and widgets in less than three years

I have a similar plan... to vacate the internet as soon as I loose interest in sex. I'm currently studying up on HTML6.

[dstiles]

May be of interest and somewhat relevant: I've just heard that G is going to give priority in the index to SSL sites. If true that should lock out a lot of good web sites. Not sure how that fits with mobile devices and non-SSL web sites.

[keyplyr]

May be of interest and somewhat relevant: I've just heard that G is going to give priority in the index to SSL sites. If true that should lock out a lot of good web sites. Not sure how that fits with mobile devices and non-SSL web sites.

This is a current list of mobile web browsers that support SSL encryption (HTTPS):

Android 1.5 +
iOS 1.0 +
Windows Phone 7 +
Microsoft Windows Mobile 5/6
Blackberry 4.3.0 +
Microsoft Windows CE 4.0
Microsoft IE Pocket PC 2003
Microsoft IE Smartphone 2003
Palm OS 5.0 +
Netfront Browser 3.0 +
KDDI Openwave 6.2.0.12 +
Brew
Opera Mini 3.0 +
Opera Mobile 6.0 +
NTT / DoCoMo
Sony Playstation Portable
Sony Playstation 3
Nintendo Wii

source: positivessl.com

RE: "that should lock out a lot of good web sites."

I don't see how any good web sites will be locked out of anything. My understanding from both the Searchengineworld and the Google Blog article is the projected advantage is currently only a .01% algo factor in ranking *if* it applies to the type of site you have, e.g. eCommerce and other sites that would require SSL. Although it does appear that Google is making a more assertive push toward the effort to secure *all* web sites by making this announcement. They've been saying this for several years.

The problem is of course, the SSL certificate industry is blatantly behind times and would need to catch up to the new demand with streamlining the certificate creation/installation/renewal process.

[wilderness]

This is a current list of mobile web browsers that support SSL encryption (HTTPS):

keyplr,
Many thanks for helping to expand my UA list ;)

Don

[lucy24]

they no longer will disclose what IPs they crawl from, to prevent whitelisting

Well, I hope they're not planning to crawl from outside the 66.249.64.0/20 neighborhood, because they'll meet a lot of locked doors :)

:: detour to last few days' logs to search for

(?<!Mozilla/5.0 \(compatible; |")Googlebot

(form that excludes regular Googlebot and Googlebot-Image) ::

Does the omission of "-Mobile" from the iPhone version (only) mean that smartphones have become so mainstream, there's no longer any reason to call yourself a Modified Googlebot?

[keyplyr]

The UA may have also been changed to determine cloaking. I think it is important to look at the header fields sent. This is enough to trigger responsive mobile mark-ups.

[wilderness]

This giving me a google-migrane ;)

[Samizdata]

Does the omission of "-Mobile" from the iPhone version (only) mean that smartphones have become so mainstream, there's no longer any reason to call yourself a Modified Googlebot?

No, it is to avoid being redirected to content made for older "feature phones".

The UA still contains the terms "iPhone" and "Mobile" and so remains a modified Googlebot.

The purpose - as stated above - is to discover smartphone-specific content (including CSS and JavaScript).

If you don't have any such content then you can probably live without the bot.

Websites generally work fine on smartphones as long as you don't mind zooming.

I haven't tried "Don's Unique Widgets", though.

I doubt that I would be allowed in.

...

[dupres01]

BBC is reporting that Google has been testing highlighting pages that have HTTPS encryption by default, and will now roll out across its algorithms.

[wilderness]

I haven't tried "Don's Unique Widgets", though.

I doubt that I would be allowed in.

Samizdata,
I have two sections of one site that allow broader access (most non-NA's are allowed) than the remainder of the site. This broader access is by design and the IP's restrictions are minimal. The UA restrictions however are based upon some early browser stuff that Jim utilized.

You'd likely get in these sections ;)

[lucy24]

and so remains a modified Googlebot

"modified Googlebot" = UA containing "Googlebot-Something" (Googlebot-Image, Googlebot-Mobile etc)
"Googlebot" = this word alone.

Turns out my log-wrangling process wasn't as narrowly constrained as I thought, so I had to go back and check. First occurrence of iPhone Googlebot

iPhone.+?Googlebot[^-]

was ... drumroll ... 3 February. Last sighting of

iPhone.+?Googlebot-Mobile

was 10 February.
So this is hardly something new.

[dupres01]

"Google's big move toward determining *true* mobile responsiveness"

How is a "true" mobile responsive site defined? I am wondering if mine qualify.

[keyplyr]

Responsive web design is a setup where the server always sends the same HTML code to all devices and CSS is used to alter the rendering of the page on the device using media queries. Our algorithms should automatically detect this setup if all Googlebot user agents are allowed to crawl the page assets (CSS, javascript, and images).

source: [developers.google.com...]

[lucy24]

I am wondering if mine qualify.

Does your CSS contain @media rules, with particular reference to max-width? If so, probably yes.

But there are two entirely different aspects to responsiveness. The visual appearance of the site, changing according to viewport width, is trivial; but that's only one part. The other is user interaction. No matter what you do, a human finger is bigger than one pixel. Paradoxically, some parts of a page have to be bigger on the smallest devices, or they just won't work.

[keyplyr]

This is what I mostly see on my sites, with no mention of "Googlebot" but the intent is obvious:

66.249.93.25 - - [12/Aug/2014:01:07:28 -0700] "GET /directory/file.jpg HTTP/1.1" 200 204157 "http://example.com/page.html" "Mozilla/5.0 (Linux; Android 4.4.2; C6903 Build/14.3.A.0.761) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.131 Mobile Safari/537.36"

[thetrasher]

There is no mention of "Googlebot" because it isn't Googlebot.
66.249.93.25 = google-proxy-66-249-93-25.google.com

More Google-Proxy: [webmasterworld.com...]

[keyplyr]

@thetrasher - There is no proof that it isn't Googlebot. Of course there's no proof that it is Googlebot either. I do know Google has been testing for mobile responsiveness as described above. There are several variations of the UA sans "Googlebot." The UA mentioned above is the perfect example of one used to test mobile responsiveness, it has all the indicators.

The same range 66.249.64.0 - 66.249.95.255, 66.249.64.0/19 is tagged as google-proxy as well as googlebot. Judging by IP range alone, there's no indication to tell either way.

Note: Blocking hits from this range that do not use the Googlebot UA may cause adverse results for sites wanting to rank in the Google mobile search index. However if your site is not designed to be mobile responsive, then why care.

[wilderness]

There is no mention of "Googlebot" because it isn't Googlebot.

There is no proof that it isn't Googlebot

66.249.79.59 - - [10/Aug/2014:01:50:17 -0600] "GET /robots.txt HTTP/1.1" 200 1157 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.59 - - [10/Aug/2014:01:50:17 -0600] "GET /MyFolder/MyPage.html HTTP/1.1" 403 647 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com
/bot.html)"

This thing grabbed pages very sparingly for six days.
Then after this request for robots.txt (not sure why as they use a different UA for iPhone) began an extensive and complete crawl, which had continued repeatedly through Friday (8/15).

It seems to have eaten enough 403s to fill its appetite.

FWIW, the standard Google Bot continues to crawl in parallel.

[wilderness]

Again, just this AM:

66.249.79.202 - - [17/Aug/2014:07:48:06 -0600] "GET /robots.txt HTTP/1.1" 200 1172 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.176 - - [17/Aug/2014:07:48:06 -0600] "GET /MyFolder/MySub/MyPage.html HTTP/1.1" 403 635 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"