Googlebot Mobile Crawls With Smartphone User Agent

[engine]

Googlebot Mobile Crawls With Smartphone User Agent [googlewebmastercentral.blogspot.com]

Today we are happy to announce that Googlebot-Mobile now crawls with a smartphone user-agent in addition to its previous feature phone user-agents. This is to increase our coverage of smartphone content and to provide a better search experience for smartphone users.

Here are the main user-agent strings that Googlebot-Mobile now uses:

Feature phones Googlebot-Mobile:

SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Smartphone Googlebot-Mobile:

Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
The content crawled by smartphone Googlebot-Mobile will be used primarily to improve the user experience on mobile search. For example, the new crawler may discover content specifically optimized to be browsed on smartphones as well as smartphone-specific redirects.

One new feature we’re also launching that uses these signals is Skip Redirect for Smartphone-Optimized Pages. When we discover a URL in our search results that redirects smartphone users to another URL serving smartphone-optimized content, we change the link target shown in the search results to point directly to the final destination URL.

[Staffa]

SAMSUNG and DoCoMo have been around for quite some time. I can't remember how far back but it could well be a year or more.

[jerednel]

I am somewhat confused by this...Could somebody clarify something for me?

Googlebot-Mobile, prior to this update, was reserved for older phones and as such they had a separate index.

Now, Googlebot-Mobile is being extended to smartphones...? Does this mean that smartphones will no longer be shown the desktop search engine results?

If so, how does their recommendation to canonical m.domain.com to domain.com mesh. I have to assume that Googlebot-Mobile will see the canonical and not index the pages appropriately.

[Mishaelo]

Does this mean that now content which is specifically designed for smartphones should have a mobile sitemap?

[g1smd]

Whereas before, Google would show URL A in all SERPs (desktop and mobile) and your site would redirect mobile users to URL M, Google now shows URL M in the mobile SERPs.

This means you now need to have a way to redirect non-mobile users requesting URL M back to URL A, or at the very least provide a link back to the normal site, because mobile users will not see and will not be aware of URL A, and may well continue to request URL M on their desktop machines.

[jerednel]

How does this work, though, if Googlebot-Mobile is crawling your mobile site and sees a rel="canonical" pointing to the main site for a given page?

This has been suggested by Google employees as a way of dealing with two websites. This previously wasn't an issue since Googlebot used the desktop version for smart phones and didn't need to index a mobile version.

So how does Googlebot-Mobile reconcile the canonical tag telling them not to index the mobile version of the site now that they need to?

[El_Matador]

i know this thread is a little old now, but as far as i can see, still no concrete answers on this. Apologies if it all seems a bit wishy washy, trying to get my head around this....

I've recently launched a mobile-friendly site. When I say mobile-friendly, i mean smartphone friendly - i'm not targetting older phones, i.e wap, just iphone and android for now.

its been set up as a subdomain off the main site, and serves completely different content to the main site. Ideally I would have liked to go down the repsonsive design route and serve 1 url for everything, but technically and creatively it would have been a nightmare.

anyway, my first question is regards to html syntax. I submitted a mobile sitemap, had it validated, and submitted it to google. Now as i mentioned, im only targetting smartphones, so im using html5 and everything renders well on the devices.

However, does this html markup mean Google does not see this as a mobile site? I ask because after submitting my mobile sitemap, the site has been crawled and indexed by the standard web bot, but yet to be indexed by the mobilebot and added to the index.

As Google mentions on this page the examples of valid mobile markup...

[support.google.com...]

And now since the introduction of the smartphone bot, I assume standard HTML will not mean a page/site is not a valid mobile site and consequently, Google not treating it as one?

I'm not using any redirects for humans or bots at the moment based on device/user-agent, so essentially at this stage im treating the two sites as two different entities, and trying to tell Google and the other bots, this is a pure mobile/smartphone site.

Any thoughts advice appreciated.

[lucy24]

Further bump:

I landed on this thread while investigating something I found while

:: insert boilerplate here ::

If I understand the original post, and linked blog item:

Googlebot-Mobile originally crawled with Samsung and DoCoMo. And then, starting in December of 2011, they added the iPhone.

Why, then, is the iPhone UA the only one I ever ever noticed until just recently?

I've only got ready access to logs going back to October 2011. Consistent iPhone and nothing else. DoCoMo made one isolated visit at the end of December (2011), and then showed up about once a month until it started crawling regularly in the second half of September (2012). Samsung showed up once in March, a few times around the end of August, and then started crawling in tandem with DoCoMo.

Each of them has made two longer visits-- one in late May, one around late August/early September-- each time picking up the identical 15 files. All 8 directories, the entire indexed content of /fun/, and one additional file in /silence/. The kind of behavior you normally see in a one-off robot that scoops and runs. Except these are all from an unimpeachable IP, 66.249.whatever-it-is.

So am I reading this whole thing backward and upside-down? Which Googlebot-Mobile came first?