Google and Mobile Phone Versions via RewriteCond

[mrguy]

I've got a site that I created a mobile version for under a directory as part of the site. The site makes use of a lot of flash and did not play well on mobiles thus the need for another version.

Google has indexed the tags from the mobile version and when you click the Google link, it takes you to the normal page but for some reason Google is displaying the mobile tags.

To redirect, I used .htaccess with the following script

RewriteEngine On
RewriteBase /

RewriteCond %{HTTP_USER_AGENT} android|avantgo|blackberry|blazer|compal|elaine|fennec|hiptop|ip(hone|od)|iris|kindle|lge\ |maemo|midp|mmp|mobile|o2|opera\ m(ob|in)i|palm(\ os)?|p(ixi|re)\/|plucker|pocket|psp|smartphone|symbian|treo|up\.(browser|link)|vodafone|wap|windows\ ce;\ (iemobile|ppc)|xiino [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(1207|6310|6590|3gso|4thp|50[1-6]i|770s|802s|a\ wa|abac|ac(er|oo|s\-)|ai(ko|rn)|al(av|ca|co)|amoi|an(ex|ny|yw)|aptu|ar(ch|go)|as(te|us)|attw|au(di|\-m|r\ |s\ )|avan|be(ck|ll|nq)|bi(lb|rd)|bl(ac|az)|br(e|v)w|bumb|bw\-(n|u)|c55\/|capi|ccwa|cdm\-|cell|chtm|cldc|cmd\-|co(mp|nd)|craw|da(it|ll|ng)|dbte|dc\-s|devi|dica|dmob|do(c|p)o|ds(12|\-d)|el(49|ai)|em(l2|ul)|er(ic|k0)|esl8|ez([4-7]0|os|wa|ze)|fetc|fly(\-|_)|g1\ u|g560|gene|gf\-5|g\-mo|go(\.w|od)|gr(ad|un)|haie|hcit|hd\-(m|p|t)|hei\-|hi(pt|ta)|hp(\ i|ip)|hs\-c|ht(c(\-|\ |_|a|g|p|s|t)|tp)|hu(aw|tc)|i\-(20|go|ma)|i230|iac(\ |\-|\/)|ibro|idea|ig01|ikom|im1k|inno|ipaq|iris|ja(t|v)a|jbro|jemu|jigs|kddi|keji|kgt(\ |\/)|klon|kpt\ |kwc\-|kyo(c|k)|le(no|xi)|lg(\ g|\/(k|l|u)|50|54|e\-|e\/|\-[a-w])|libw|lynx|m1\-w|m3ga|m50\/|ma(te|ui|xo)|mc(01|21|ca)|m\-cr|me(di|rc|ri)|mi(o8|oa|ts)|mmef|mo(01|02|bi|de|do|t(\-|\ |o|v)|zz)|mt(50|p1|v\ )|mwbp|mywa|n10[0-2]|n20[2-3]|n30(0|2)|n50(0|2|5)|n7(0(0|1)|10)|ne((c|m)\-|on|tf|wf|wg|wt)|nok(6|i)|nzph|o2im|op(ti|wv)|oran|owg1|p800|pan(a|d|t)|pdxg|pg(13|\-([1-8]|c))|phil|pire|pl(ay|uc)|pn\-2|po(ck|rt|se)|prox|psio|pt\-g|qa\-a|qc(07|12|21|32|60|\-[2-7]|i\-)|qtek|r380|r600|raks|rim9|ro(ve|zo)|s55\/|sa(ge|ma|mm|ms|ny|va)|sc(01|h\-|oo|p\-)|sdk\/|se(c(\-|0|1)|47|mc|nd|ri)|sgh\-|shar|sie(\-|m)|sk\-0|sl(45|id)|sm(al|ar|b3|it|t5)|so(ft|ny)|sp(01|h\-|v\-|v\ )|sy(01|mb)|t2(18|50)|t6(00|10|18)|ta(gt|lk)|tcl\-|tdg\-|tel(i|m)|tim\-|t\-mo|to(pl|sh)|ts(70|m\-|m3|m5)|tx\-9|up(\.b|g1|si)|utst|v400|v750|veri|vi(rg|te)|vk(40|5[0-3]|\-v)|vm40|voda|vulc|vx(52|53|60|61|70|80|81|83|85|98)|w3c(\-|\ )|webc|whit|wi(g\ |nc|nw)|wmlb|wonu|x700|xda(\-|2|g)|yas\-|your|zeto|zte\-) [NC]
RewriteRule ^$ http://www.example.com/mobileweb [R,L]

Any idea why google is identifying itself as mobile user agent and indexing the mobile version?

[edited by: tedster at 10:37 pm (utc) on Oct 15, 2010]

[jdMorgan]

Because it wants to discover mobile content as well as desktop/laptop content.

Now that it has discovered your /mobileweb URLs it will remeber them 'forever,' and the only way to get those URLs out of the SERPs would be to permanently redirect Googlebot (only) from /mobileweb URLs to / URLs, and also to fix the rule above by specifically excluding "Googlebot-Mobile" user-agent strings, e.g. add

 RewriteCond %{HTTP_USER_AGENT} !Googlebot 

at the top of your RewriteConds so that Gbot won't ever get redirected to your mobile content.

You could also significantly reduce the number of subpatterns in your list and get rid of many of the short, ambiguous (and therefore potentially dangerous) ones by looking for "common" stuff, like

 ^([^\ ]+\ )+[Pp]rofile/MIDP-[1-9]\.[0-9]\ [Cc]onfiguration/CLDC-[1-9]\.[0-9] 

and mobile operating system strings (e.g. "Symbian"), instead of constantly trying to catch each model as it comes out. Any phone UA string already covered by one of the "common" strings then won't need to be added to the list, or could be removed if already listed.

Jim

[mrguy]

Thanks Jd,

Would I create an .htaccess file for the mobile folder and place the following in it to direct googlebot back to the correct area:

RewriteEngine On
RewriteBase /

RewriteCond %{HTTP_USER_AGENT} Googlebot [NC]

RewriteRule ^$ http://www.example.com/ [R=301,L]

[jdMorgan]

No, you can't go redirecting back and forth, you'll get a loop.

Put the negative-match RewriteCond I posted above at the top of your existing rule, then wait a week or so. After you've seen Gbot NOT getting redirected to your mobile site several times, then you can put another rule or rules -- either in your mobile subdirectory or in your main .htaccess file -- to redirect any further direct (that is, unredirected) Gbot requests for /mobileweb URLs back to the corresponding non-mobile URLs.

The only difference between putting this second rule in /mobileweb/.htaccess vs. /.htaccess is that in /.htaccess, the pattern would be ^mobileweb/?$ instead of ^$ in /mobileweb/.htaccess

Jim