Mobile friendly - vary http header question

[MHes]

Hi

I have made my site more mobile phone friendly by detecting mobile users on the fly and serving a different layout (via triggering a different stylesheet and building the page without tables).

Google recommends that by make a site that, in their words, 'Dynamically serving different HTML on the same URL' I should change the server http header to be a " Vary HTTP header"

They say, ...
"It helps Googlebot discover your mobile-optimized content faster, as a valid Vary HTTP header is one of the signals we may use to crawl URLs that serve mobile-optimized content."

It has been suggested that if some of the pages on my site change for mobile users, but some don't, google will not like me having a " Vary HTTP header"

Anybody got any experience of this. There are some pages that rarely get visited and to make them mobile friendly is a lot of extra work!

[deadsea]

HTTP headers can be set on a page level basis rather than for the site as a whole. If some of your pages vary based on user agent, you can set that header just for those pages.

[MHes]

Many thanks Deadsea. I have many generated pages (2.3 million indexed) that will change depending on the user agent, and about 80 that won't. These pages that won't change are rarely visited but important for the spider to navigate my site.

Someone suggested I generate the headers via the perl scripts that generate the pages on the fly, but I have no idea how to do this! The safer option seems to be to set the Vary http header on the server for all the pages. I know you are right about being able to set them for specific pages, but there are too many to list..... or am I missing something? (Way out of my depth on this one!)

[deadsea]

I don't know what perl modules you are using, but the code to send a header should be quite simple. You probably have some code to set the Content-Type header somewhere already. You could add a line right next to it to set the vary header.

If you are using CGI.pm it might look something like this:

print $cgi->header( -type => 'text/html;charset=UTF-*', -vary => 'User-Agent' );

If you want advice specific to your situation, you'd have to say what type of perl modules you are using.

[MHes]

Hi Deadsea

Yes, I use CGI.pm and have....

# Send the MIME header. Double new line!
print "Content-type: text/html\n\n";

Do you think a simple.....

print "Content-type: text/html Vary: accept-encoding \n\n";

would work?

Presumably I then put that on the pages that might vary due to user agent. The other pages, that are just static html, where do they get the header from? I am assuming that the server generates a header and this will not be a problem for the dynamic pages. In other words, I don't get two headers generated for the dynamic pages, one from the code above and another from the server.

Sorry if I am muddled but hopefully this will be useful to others as well!

[deadsea]

It needs to be like this. Single new lines between headers, double newline after the last one.

print "Content-type: text/html\n";
print "Vary: User-Agent\n\n";

The double new line needs to be after the last header. I believe that you are varying the pages based on the user agent for your mobile site, not based on accept-encoding, so your vary header should be "User-Agent".

[MHes]

Deadsea - Many thanks for your help.

Everything is now Vary: User-Agent as google suggested. It will be interesting to see if making the site mobile friendly has knock on effects with overall ranking.

[klark0]

@MHes, it probably won't. But when I finally got my Vary header sorted with a "Same URL, Different HTML" setup, Googlebot Mobile bot re-crawled the entire site that same day.

[wenwon]

Thanks for the previous answers on this page.

I have a question from a layman's perspective. With regards to the vary HTTP header. Does anyone know the exact code (for the vary HTTP header) that is supposed to be placed within the HTML of a web page?

Google seems to indicate that this is the code that goes on each page of a site that offers a redirection option:

GET /page-1 HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.133 Mobile Safari/535.19
(...rest of HTTP request headers...)

HTTP/1.1 200 OK
Content-Type: text/html
Vary: User-Agent
Content-Length: 5710
(... rest of HTTP response headers...)

I'm getting conflicting responses from my server people and my programmer with regards to the above code.

Is this the right code that is listed on every page? If not, can someone point me in the right direction?

[lucy24]

Short answer: It doesn't go in the HTML at all. It goes in whatever generated the HTML. This might be either php/cgi/you-name it, or it might be individual human effort.

[wenwon]

How does that work for a website that has over 1000 static html pages? I'm sorry for my lack of ignorance but I'm not completely understanding your statement.

[lucy24]

You may be tripping over the multiple meanings of the word "head(er|ing)". The request and response headers exchanged by the user-agent and your server are one thing. The "head" section of a page is another.

If you've got static html pages then the header question probably doesn't apply. Responsive pages that work within html and css-- constructions like

@media screen and (max-width: 960px)

or

<link rel = "stylesheet" type = "text/css" media = "screen and (max-width:960px)" ...

leading to different stylesheets or different lines in the same css-- don't need any more information. Everything the search engine needs to know is right there in the html.

If you have user-agent detection in your htaccess and send different users to different pages, it may or may not be appropriate to use a header declaration. Personally I wouldn't bother. Save the "vary" business for pages that are constructed on the fly.

That's what I meant by "whatever generated the html". If the pages are hand-rolled by you, then the <header> segment will be similarly hand-rolled. If it's page-specific, it will go either inside a <Files> envelope or in the htaccess belonging to a subdirectory.

[Robert Charlton]

To further clarify when "The Vary HTTP Header" applies, looking at it from a different angle. Google is not necessarily recommending it for all mobile sites.

The article partially quoted in the opening post here is a Google Developers >> Webmasters article...

Building Smartphone-Optimized Websites
Last updated April 12, 2013
https://developers.google.com/webmasters/smartphone-sites/details [developers.google.com]

The article begins...

Google supports smartphone-optimized sites in three configurations

1 - Google's recommended configuration is "responsive web design... sites using the same set of URLs with the same HTML to all devices... and using CSS to change rendering.

2 - Sites dynamically serving all devices on the same set of URLs, but "each URL" serves different HTML and CSS, depending on the user agent. This is the configuration for which Vary HTTP header applies.

3 - Sites having separate mobile and desktop URLs.

Under "Dynamically serving different HTML on the same URL", the article then goes on to highlight the two important implications of the Vary HTTP header, which are that...

1 - It signals caching servers used in ISPs and elsewhere to consider the user agent...
2 - It helps signal to Google that alternate mobile content is being served.

So, be aware that not every site configuration needs to be concerned about the Vary HTTP header. That said, as large screen mobile hardware becomes available... larger smart phones, tablets, convertible laptops, etc... viewport size becomes a key consideration, and there is ongoing discussion about whether responsive design or alternate content might be preferable.

Recent discussions (to choose among several)....

Checklist for Mobile Site Optimization?
Dec 26, 2012
http://www.webmasterworld.com/google/4531372.htm [webmasterworld.com]

How Do We Do This?
Want to take a version of our PC site to Mobile
Dec 3, 2012
http://www.webmasterworld.com/pda_mobile_computing/4524592.htm [webmasterworld.com]

Responsive Design versus separate site
Feb 24, 2013
http://www.webmasterworld.com/pda_mobile_computing/4548452.htm [webmasterworld.com]

[wenwon]

Robert. Thanks for your insightful response.

I certainly understand your point of view. I've showed that Google developer URL that you speak of to 5 developers today. I literally got 5 conflicting answers. Most had never heard of the vary HTTP header.

I'm a proponent of a separate mobile site due to the many configuration issues across unique devices we encountered while trying to create a responsive design.

My configuration will be simple, the normal website will serve a URL like this for desktop devices:

www.example.com/this_is_a_test.html

If the person visits using a mobile device, they will see this URL:

www.example.com/mobile/this_is_a_test.html

Under your theory, do you believe that our particular situation warrants the use of the Vary HTTP header?

Again, thanks for your previous response. Literally, it's the only thing that has made sense to me in speaking to many professionals and searching online.

[lucy24]

Under "Dynamically serving different HTML on the same URL", the article then goes on to highlight the two important implications of the Vary HTTP header, which are that...

If you are redirecting, the URLs are by definition* different. The "Vary" header is only important to search engines when you're serving different content from the same URL. That means the written HTML, not what the user's eyeballs see.

In htaccess, headers look like this (here quoting the popular "don't index my bleepin' robots.txt" header):

<FilesMatch "\.(js|txt|xml)$">
Header set X-Robots-Tag "noindex"
</FilesMatch>

Or, in apache-speak-- with the parts I actually used shown in boldface--

Header [condition] set|append|merge|add|unset|echo|edit header [value] [replacement] [early|env=[!]variable]

"header" (the second occurrence) is the name of the header, here "Vary". "value" is what you want the header to say; this part is optional. The obligatory parts are only #1 what you want to do with the header-- set, unset, add etc-- and #2 which header you're talking about.


* By definition of the word "redirect", that is.

[Robert Charlton]

Overlap with my writing and lucy24's posting...

wenwon, thanks for your follow-up question.

Under your theory, do you believe that our particular situation warrants the use of the Vary HTTP header?

Yes, it probably does... and my apologies for not noting the particular situation where it would be warranted. In Google's article, that would be the situation you describe, configuration #3, the Separate mobile URLs... assuming that you will choose some sort of User-Agent Detection to redirect your site.

As described in the very last section of the article...

Redirects and the HTTP Vary header

Please note that if your site automatically redirects mobile visitors coming to the desktop site to the mobile site, or vice versa, please be sure to configure your server to include the Vary HTTP header as described on this page [developers.google.com].

To make a separate note of the page, it is...

Redirects and User-Agent Detection
Last updated April 12, 2013
https://developers.google.com/webmasters/smartphone-sites/redirects [developers.google.com]

The Redirection article also emphasizes the need for Vary HTTP headers with automatic redirects, for reasons already discussed... caching and Googlebot.

Under Supported redirection techniques, the article discusses both...
- HTTP redirection
- JavaScript redirects

I have no time now to get into whether you should try for responsive design, or to implement another approach, but I do feel that you should implement some sort of automatic redirection related to viewport... ie, you should not rely on users deciding they don't like your static site and clicking over.

I myself would probably recommend different approaches for a new site than for an old site. The approach or combination of approaches you choose may depend on the degree of interactivity of your site. Definitely follow up the technologies that coopster suggests, which are probably more cutting edge than I think you're prepared to try... but there are some points made that you definitely should read.

One comment I should make now, though. You mention this URL setup for mobile...

http://www.example.com/mobile/this_is_a_test.html

IMO, this is going to be hopelessly confusing to implement, unless you are defaulting to your static site for the home page and then relying on manual choice, not IMO a good approach.

Though Google indicates no preferences, you're going to find it much easier to have a one-to-one correspondence between desktop and mobile pages by using a mobile subdomain. So you might have...

- for desktop, either
http://www.example.com/this_is_a_test.html
or
http://example.com/this_is_a_test.html

- and for mobile
http://m.example.com/this_is_a_test.html

If you can't do this easily at your present host, I recommend that you get a new DNS and hosting setup before you even consider trying to offer a mobile site. The additional control at the host level is something you will find necessary as you move ahead.

Regarding what "headers" are, here's an article that I've cited in this forum several times. It's one of the few articles I've seen that even tries to discuss the http protocol and how the web works in layman's terms....

How the web works: HTTP and CGI explained
[garshol.priv.no...]

[Robert Charlton]

I'm noting that the Redirection article doesn't get into the question of URL change and caching, as lucy does. Lucy's point about URL caching and the definition of redirection makes sense.

I have no axe to grind here... it may be that the second Google article glosses over that point, or that I've misworded the explanation, or it may be that Google has reason for wanting HTTP Vary Header to be used situation #3 as well... which does involve a URL change, and which is how I'm reading it.

In any case, I currently have no more time to give to the topic, but I'd love to see a resolution of the questions.

[lucy24]

Please note that if your site automatically redirects mobile visitors coming to the desktop site to the mobile site, or vice versa, please be sure to configure your server to include the Vary HTTP header

Oops. I entirely missed that part, which means that about three-quarters of my previous post is wrong.

Did I say anything about caching? I think that was a different thread.

:: detour to (re)read google article ::

A common issue is redirecting users to irrelevant pages. For example, if a user visits example.com/article29, your website should redirect them to the equivalent mobile-optimized page, such as m.example.com/article29, instead of to the m.example.com homepage. This makes sure that incoming links and users' bookmarks work across devices
....
Also, we suggest giving users a way to override the redirect policy

I'll be darned. Just when you think everything g### says is utterly wrong, self-serving and can be safely ignored, they come out with something sensible. (Which is to say, something that people hereabouts have been saying too.)

we do not have any preference and recommend that webmasters consider their users

:: sitting on hands ::

They do seem to be saying that a redirect based on UA detection merits a Vary header. Wouldn't you think the robots could figure it out for themselves? All three Googlebot-Mobile versions use a pretty standard UA, the kind an ordinary RewriteRule would recognize.

[Robert Charlton]

Did I say anything about caching? I think that was a different thread.

I was looking for the merits of what you were saying, Lucy, and caching was an implication I drew from this particular comment you made....

The "Vary" header is only important to search engines when you're serving different content from the same URL.

Google suggests that Vary Headers will alert ISPs etc who might otherwise serve cached pages to "consider the user agent when deciding whether to serve the page from cache or not." (Having the correct page served, of course, also indirectly helps Google.)

But, if the URL changes, as with the mobile subdomain added, there would be no reason for an ISP to serve the cached page. They would be dealing with a different URL. Thus, I felt that you might have had caching in mind (among other considerations) when you made your comment about the importance of the "Vary" header.

[wenwon]

The placement of this sentence is probably the main reason why I received 5 differing opinions yesterday on how to proceed:

Please note that if your site automatically redirects mobile visitors coming to the desktop site to the mobile site, or vice versa, please be sure to configure your server to include the Vary HTTP header

The way in which Google has presented this information can illicit a broad range of opinions on how to move forward. For someone who is not technical , it makes it incredibly difficult to understand what the best course of action would be. The Google Product Forums are a virtual "ghost town" with regards to any subjects related to this.

@lucy24, Robert Charlton: I really appreciate both of your efforts to help explain this. It's been very helpful.
.

[edited by: Robert_Charlton at 5:55 pm (utc) on May 3, 2013]

[Robert Charlton]

To follow up on the above...

I see that Matt Cutts has discussed the Vary Header in a recent video. Some individual Content Delivery Networks like Akamai don't support full use of Vary HTTP header and won't automatically cache a URL carrying the Vary header "if it's only by user agent". In the video, Matt explains the importance of caching, explains also why Akamai might not want to cache, and also explains benefits of using the Vary header anyway... that the Vary header may be helpful in other ways and be a key indicator to Google in how to index the page....

Should I use the Vary HTTP header on URLs that redirect based on user-agent?
trt 4:37
You Tube - May 8, 2013
http://www.youtube.com/watch?v=va6qtaiZRHg [youtube.com]

It's an interesting video, even though, as Matt says, it's pretty esoteric. Key point (and one that I've got to confess has troubled me as the number of possible user agents continues to increase) is, as Matt says...

...there's a lot of different user agents, and I can certainly understand if somebody like Akamai would say, 'well, if there's 600 user agents, we don't want to save 600 copies of this page'....

Matt recommends that people continue to use the Vary Header because of all the possible other caches along the way, and because it helps Google determine that you've got different desktop and mobile content....

My emphasis...

...it's still a good idea, if you have a website, to try to send that information along so that the various caches between you and the user can try to do something smart.

...in terms of Google indexing, it's not apparent to us whether a URL can return different content based on the user agent, and so we actually do look at the HTTP Vary header in order to help us assess whether a page might be targeted both to desktop and to mobile.

...it's kind of a complicated issue.

[burnseo]

It has been suggested that if some of the pages on my site change for mobile users, but some don't, google will not like me having a " Vary HTTP header"

does anyone know if search engines will not like having a Vary HTTP Header on pages that don't have a mobile version? - It seems that it's possible to implement on a page level, but after reading the comments I'm still unsure if there is any real problem around doing it as a global command. Sorry if I missed something in the comments. I'm just trying to understand a statement in the original comment better.

[phranque]

I think google would assume the other version simply hadn't yet been discovered.