Googlebot-Mobile - how does it index / how should we design our site?

[nquinn]

We are in the process of designing a new mobile site for our company, and are torn between two possible options.

The end goal is to support desktop web, smartphones, and feature phones.

Option 1: Two mobile sites - one for smartphones, the other for feature phones.

The idea is that our regular web pages would use javascript and css3 media queries on the client side to detect what features are available per device.

iPhone and android users would be served the best experience possible.

We would probably use server-side user agent detection to push all known feature phones to an alternative subdomain such as m.domain.com.

Option 2: One mobile site. Similar to above, but no server-side user agent detection at all. We would somehow detect that feature phones are present and dumb down the experience.

What I know:
- Googlebot serves smartphone users (iphone, android) the regular index of websites, not sites from their mobile specific index. The idea is that these phones are capable of viewing the full site, so they don't want to dumb down the experience.
- Googlebot-mobile indexes only sites that are mobile friendly. It seems to identify them by their DTD (cHTML, etc). Other than that there is not much information.

Specific questions that I need help with:

1. If we provide only one URL for all devices, will it still be crawled by both Googlebot and Googlebot-mobile? My concern is that our pages will fall out of the "mobile index".

2. How does Googlebot-mobile determine a site should be in the index? We are trying to avoid using user-agent detection and a second website, but this means when Googlebot-mobile hits our www. page I'm not sure what we will end up serving it because we plan to use client side detection.

3. We'd like to ideally build our site using HTML5 and parse JSON calls. However, much of our data will be hidden behind Javascript so Google won't see it. Any suggestions?

4. We originally planned to use rel=canonical on the smartphone version of the site to point to desktop web, but this would ruin the indexing of feature phone url's. Thoughts?

In short, we'd like to present our site at 3 different levels: full desktop web, smartphone mobile, and feature phone mobile. At the same time, we'd like to maintain our feature phone mobile pages in Google's mobile index (as crawled by Googlebot-mobile)

Anyone else have tips on this?

I've already read this [ [googlewebmastercentral.blogspot.com...] ] , but there is still lots left unsaid.

[tedster]

There is a serious learning curve in doing mobile well. I know enough about that to outsource the development to someone who is focused in the area of mobile SEO.

That said, at Pubcon I picked up one tip - pages that are too big will not be shown in Google's mobile search for feature phones. I didn't make note of the exact size, but definitely keep the content light for feature phones. That makes plenty o' sense to me.

Also, John Mueller has a Google Plus page devoted to the fine points of Mobile Websites & Google's crawling, indexing and ranking - [plus.google.com...]

[edited by: tedster at 10:56 pm (utc) on Nov 17, 2011]

[deadsea]

I have worked with sites with two versions, one for the desktop and one for smart phones. I haven't tried adding in a site for non-smart phones. I haven't seen numbers that enough people access web pages on these devices to convince me that it is worth it to have a version of the site for them.

1. If we provide only one URL for all devices, will it still be crawled by both Googlebot and Googlebot-mobile? My concern is that our pages will fall out of the "mobile index".

I have worked with two sites that have one url but provide different content to desktop and smart phone users. This is a great way of doing it. The urls will be crawled by both Googlebot and Googlebot-mobile.

2. How does Googlebot-mobile determine a site should be in the index? We are trying to avoid using user-agent detection and a second website, but this means when Googlebot-mobile hits our www. page I'm not sure what we will end up serving it because we plan to use client side detection.

User agent detection is the way to go. There are user agent lists that can be imported and in my experience work very well. Just make sure that googlebot-mobile is included in the mobile experience. According to official Google documentation user agent sniffing for this purpose is NOT cloaking as long as mobile user and googlebot-mobile get the same content; and googlebot and desktop users get the same content.

3. We'd like to ideally build our site using HTML5 and parse JSON calls. However, much of our data will be hidden behind Javascript so Google won't see it. Any suggestions?

Googlebot has started crawling and indexing javascript recently to some extent, however my advice would be that you shouldn't do it this way if you want to get search engine traffic.

4. We originally planned to use rel=canonical on the smartphone version of the site to point to desktop web, but this would ruin the indexing of feature phone url's. Thoughts?

If you have two different sets of urls, then you should sniff user agents and issue 301 redirects to get users to the correct site. You also might want to allow users to view the full site on their phone or the mobile site on their desktop, perhaps with the use of cookies.

[scooterdude]

brilliant question, I'd been trying to frame this question but , this addresses most things , cheers

iwas a bit surprised tby the percentage of mobile users when I checked it in the analytics

Note:
I see that some sites like amazon pass my mobile device straight to the mobile site, apparently on the same url:frustrated users is a concern here

other sites, offer the option of switching to a mobile site on a different url:duplicate content is a concern here

I am keen on combining the approaches

[deadsea]

Amazon, like many mobile sites has a "full site" link at the bottom of the mobile pages.

[netmeg]

I'm running 30-40% mobile traffic on a bunch of my event sites. The first year I wanted to serve mobile, I put another site on m.example.com. This past year, since everything was converted to WordPress anyway, I went with a premium mobile plugin (using user agent detection) No search engine issues at all, and there's a clear link at the bottom for people who prefer the "classic" view. I'm happy with it. And it's indexed just the same as it always was.

[Marshall]

I'll be the first to admit that I don't research a lot of these tricks and tips as much as many of you, and I'm grateful for those that do. But why can't they come up with a simple meta tag indicating what type of display a page is primarily made for, e.g.

<meta type="display" content="handheld" /> or

<meta type="display" content="screen,handheld"> this either indicating both or order of best display.

You can specify CSS, so why not the page itself.

Just a thought.

[netmeg]

Well I don't know about the tags, but you might want to look at responsive design. My next set of themes for my sites are all going to be responsive design - this is where it detects the screen size / viewport, and reorganizes itself accordingly. A great example is the current Boston Globe site - if you play around with resizing the browser window, you'll what I'm talking about. It's *way* cool.

[Marshall]

netmeg,

My designs are fluid and actually display well regardless of the device. My tag suggestion was more for the benefit of indexing web vs. mobile.

[netmeg]

But... it's the same indexing. No need to do it separately for "web" and "mobile"

[scooterdude]

well no, what I want to know is this:

Do the crawlers from the search engines present themselves as mobile devices thereby triggering a mobile offering from the site

afterall, if the mobile crawler has the same character as the standard crawler, it will simply get the same page served up

[deadsea]

Google has a crawler named googlebot-mobile. It presents itself as mobile so that it will get mobile pages.

[scooterdude]

actually google-mobile does require some more research , some folk have had unintended outcomes