WMT verification... Google crawling with GeoIP?

[Shai]

Hi All, Been browsing this site for years now and finally decided it’s time to pay my dues. So firstly, thanks to everyone here for all of the information and guidance you have all provided.

I have a strange problem which I previously never came across, and, the fact that there are several companies controlling different aspects of this project, it makes things slightly more difficult to crack.

We have a client that has two sites. Example.co.uk and example.ie. The way its set up is that the .ie is for Ireland and the rest of the world, and the .co.uk is for uk only. Traffic is directed via GeoIP. We have not been consulted in this arrangement and it’s not how I would do it. In fact, it looks quite a strange arrangement to me, but I have no say in the matter.

I am only looking after the .co.uk site.

The problem I have is that I am unable to get the site verified in WMT. I have tried three methods.

1.Upload the html file. All fine, but as the Google verification server is obviously being redirected to the .ie site, it is not finding the correct file.
2.Meat Tag: as above.
3.DNS TXT record. This is a strange one as I would have expected this to work. The company in charge of the domain says they have followed the instructions I sent to the letter and have created a TXT record with the verification code. However, using a lookup tool, I am unable to see this record. They are insistent that the record exists. Is there any way the GeoIP redirection is also causing this problem? I would have thought that as this is a domain lookup it would have nothing to do with the redirection? Anyway, it’s also failing.

This whole problem has also made me think about the a possible problem with Google not crawling the .co.uk site as it will always be redirected to the .ie site. Is there any way round this?

[Andy Langton]

Google always crawls from the US, so if you redirect those visitors, you'll be redirecting Google also. If you make an exception just for Google, strictly speaking that would be cloaking (and could be detected, since Google's anti-cloaking checks come from the US also).

GEO-IP has nothing to do with the DNS - probably the record is not present or is not set correctly.

[Shai]

So, even if I get the verification to work, I am still left with a problem right? Would you expect the current number of the .co.uk pages indexed to just start to drop over time as google is no longer able to crawl them?

Is there any way round this? Or do I just tell the client to buy the .com and have a example.com/en. and example.com/ie/ directories?

Thanks for your help.

[Andy Langton]

If you redirect US visitors from .co.uk somewhere else, you won't get any crawling of those URLs (or any indexing) and thus no rankings for UK search.

Using ccTLDs like .co.uk is the right approach (and a much better approach than subdirectories). The current geo-targeting process is wrong.

If someone directly navigates to a .co.uk domain, they should see UK content and should not be redirected. Use a .us or .com if you want US/worldwide targeting, and redirect (for instance) UK visitors from the .com to a .co.uk.

Search engine infrastructure is US-centric, basically, so you cannot get geo-redirects to work any other way.

[ColourOfSpring]

Might be that the Time To Live (TTL) for the DNS records of this domain is something a little lengthy - don't know how long you've waited but if you can't see the DNS changes via your own lookup, might be they will just take time to propagate around the internet.

[lucy24]

This whole problem has also made me think about the a possible problem with Google not crawling the .co.uk site as it will always be redirected to the .ie site. Is there any way round this?

This issue comes up a lot, but for some reason the question is usually about Pakistan [webmasterworld.com]. If someone knows the reason, don't tell me. I'm tolerably certain I don't want to know.

Where does the redirect happen? On the page itself, in some preliminary php, in an Apache mod-- where?

I'm thinking something along the lines of

<Files "google-blahblah-exact-name.html">
{stuff to prevent redirection here}
</Files>

but the exact content of the envelope depends on how the redirect is done.

I don't think code that applies purely to site authentication can be considered cloaking, can it? afaik, not even google has started indexing its own authentication files

[phranque]

welcome to WebmasterWorld, Shai!

So, even if I get the verification to work, I am still left with a problem right? Would you expect the current number of the .co.uk pages indexed to just start to drop over time as google is no longer able to crawl them?

yes you will still have a problem if google can't crawl the .co.uk urls.
are any .co.uk urls indexed? (using a site:example.co.uk search)
are they in the google.co.uk index and/or the google.com index?

Is there any way round this? Or do I just tell the client to buy the .com and have a example.com/en. and example.com/ie/ directories?

ideally the client would keep example.ie and example.co.uk for the country-code specific regions for which they are intended, and also use example.com to serve content for ROW (the "rest of the world").
continuing ideally, these domains would each contain unique content based on the intended audiences.
then host the example.ie and example.co.uk domains within their intended countries and host the example.com domain in the US.
but first fix the automatic redirects based on visitor location - your problem is described in the blog post linked below.

Official Google Webmaster Central Blog: Working with multilingual websites:
http://googlewebmastercentral.blogspot.com/2010/03/working-with-multilingual-websites.html [googlewebmastercentral.blogspot.com]

If you’re going to localize, make it easy for Googlebot to crawl all language versions of your site. Consider cross-linking page by page. In other words, you can provide links between pages with the same content in different languages. This can also be very helpful to your users. Following our previous example, let’s suppose that a French speaker happens to land on http://example.ca/en/mountain-bikes.html; now, with one click he can get to http://example.ca/fr/vélo-de-montagne.html where he can view the same content in French.

To make all of your site's content more crawlable, avoid automatic redirections based on the user's perceived language. These redirections could prevent users (and search engines) from viewing all the versions of your site.

in your case the language is irrelevant but the technical issues are similar.

you should also read this blog post to understand the issues.

Official Google Webmaster Central Blog: Working with multi-regional websites:
http://googlewebmastercentral.blogspot.com/2010/03/working-with-multi-regional-websites.html [googlewebmastercentral.blogspot.com]

you might consider using the link rel alternate hreflang element to cross-reference the various versions as described in the blog post linked below and mentioned in the quote above, but you'll still have to fix the redirect problem so that googlebot can request the other linked versions of each page.

Official Google Webmaster Central Blog: New markup for multilingual content:
http://googlewebmastercentral.blogspot.com/2011/12/new-markup-for-multilingual-content.html [googlewebmastercentral.blogspot.com]

[Shai]

Just wanted to come back and update this thread.

We have now removed the GeoIP redirects and have setup each site Geo targeting in WMT correctly. We are also going through the site and changing the product text slightly to reduce the chance of duplicate content. The site was dropping quite a few pages from the index on a daily basis before the removal of GeoIP but now started to grow again so it’s obviously working. Will upload a new sitemap today to speed things up in this respect.

The only problem we have is that both sites are on the same UK dedicated server. I will keep it like this for now and see how Google behaves. To compensate, I’ll try and increase Geo signals such as clear address in the footer of each site.

Thanks again for everyone’s input. Its very much appreciated.

[phranque]

thanks for the update, Shai!

[sdgit]

I have a situation similar to Shai's, and wonder if I could ask for everyone's input. New user here, apologies if this should be a new thread.

A client wants to divide their domain into country-based subdirectories such as domain.com/uk, domain.com/au, domain.com/fr, etc. Each subdir will have totally unique local content, not just translations. The root domain.com will be for the US and ROW.

They also want GeoIP so that, for example, users in the UK will be forwarded from domain.com (or any non-UK subdirectory) to domain.com/uk. As I understand, this is primarily for regulatory compliance, and they can't just let users choose which site they want to use.

Unfortunately the client has no budget to purchase ccTLDs (instead of subdirectories) for the country-specific pages. Subdomains are possible but they prefer subdirectories because they pass authority to the root domain.

Their goal is for each local subdirectory to rank highly in local search, similar to Apple.com, Apple.com/uk, etc.

Is this subdirectory-GeoIP approach workable from a search perspective, or fatally flawed? Will penalties to subdirectories strongly affect the whole domain?

Thanks in advance for any help.

[aakk9999]

Welcome to WebmasterWorld, sdgit!

Geo-targeting by subdirectory is pretty standard practice and easily set up. It only works if the main domain is gTLD, which, from your example, yours seem to be. I have used this techinque on a number of sites with no problems whatsoever.

What you need to do is go to WMT of your example.com site and add each subdomain as a "separate" site. I.e. you go and "Add a new site" and then in the URL name you type example.com/uk/ for the UK folder or example.com/au/ for Australia folder and so on.

Once you add folders in this way to WMT, go to manage each of these folder-based "websites" and set up geo-targeting for each as you wish.

Subdomains are possible but they prefer subdirectories because they pass authority to the root domain.

Yes, this is my experience too.

Will penalties to subdirectories strongly affect the whole domain?

They may or they may not, depending on the penalty. I think that from this perspective, the subdirectory is just a "page" on the domain. I have not heard of a site penalised based on "one directory", e.g. situation where "all URLs with /au/ are penalised".

But obviously, if the domain is penalised, it will affect all subfolders.

[sdgit]

Thanks aakk9999 for the welcome and quick reply!

Here is a wrinkle: In the interest of speed and SEO, each subdirectory would be hosted on a local server, and GeoIP would resolve DNS for UK traffic to domain.com/uk.

However while managed DNS providers do offer such a GeoIP service, traffic can only be sent to subdomains (e.g. uk.domain.com), not subdirectories (domain.com/uk).

One solution might be to resolve DNS to uk.domain.com using GeoIP, then server-side redirect to the domain.com/uk folder, located on the same server.

So the questions are:
- Would this work?
- How would page load times be affected by the redirect?
- Would Google index the subdomain, or only the subfolder?

[aakk9999]

Let me see if I understood this correctly:

- You want to host each country's content in the particular country
- You can only do this if you go for subdomain geotargeting model
- But you want to have folder based geotargeting model

So, to overcome this, you are thinking of:
- hosting subdomains in each target country
- redirecting visitors using 301 redirect from each subdomain to the main domain that is folder based

The reason (your objectives) for doing this is:
- speed (as you say IP is local)
- having an IP that is local of the country for SEO purpose

Unless I have misunderstood your proposed setup - I would NOT do this.

And if you do this, you will not meet your objectives anyway.

1) Speed... it will not be faster. Firstly, you are going over 301 redirect. So it is 2 requests, not one. Further, the second request (the one that is executed as a result of 301 redirect) goes to the www.example.com domain anyway wherever it is hosted

2) Local IP address... is this because of Google geolocating sites based on IP? Whilst Google does look at the IP geo-location to try to determine how the site should be geo-located, this is done for sites that are not geolocated in WMT, i.e. for gTLD domains that are not geolocated. If you geolocate a directory to a country, Google will take geolocation from this.

To be honest I have never heard of this kind of setup and I am not sure what would really happen in SERPs. I presume Googlebot would also go via redirect. Do not forget that by going via redirect you lose approximately 15% of link juice.

Server setup is not my strong point however, and I may not be understanding properly your setup.

[sdgit]

Great insights, thank you aakk9999.

You are correct about the proposed setup. The motivation for the 301 redirect and local IP address was speed - to allow for managed DNS (faster DNS resolution, DDoS mitigation) and load times. But your response makes clear that a redirect would erase any speed gained and cost that 15% link juice you mention.

One alternative might be to use reverse proxy at the primary host, so each country folder can be located on a separate server but still appear in the same URL and pass authority to the root domain. This setup implies DNS resolving to the primary host's IP, and then use a GeoIP service to send visitors to the appropriate folders. But this strategy is also 2 steps, so I wonder if it's equally slow?

[phranque]

i would suggest using subdomains and linking among the subdomains exactly as you would if you were using subdirectories.
use GWT manual geotargeting to specify the region you are targeting with each subdomain.
google considers GWT geotargeting as an equal strength signal to ccTLDs.

the second strongest signal is server location so i would host as many subdomains in-country as is practical.

They also want GeoIP so that, for example, users in the UK will be forwarded from domain.com (or any non-UK subdirectory) to domain.com/uk. As I understand, this is primarily for regulatory compliance, and they can't just let users choose which site they want to use.

how are you going to account for the foreign traveler or ex-pat situation?

[aakk9999]

But this strategy is also 2 steps, so I wonder if it's equally slow?

As I said I am not an expert here, but I think 2-step will be slower as you have an extra step (#1) whereas #2 has to be executed one way or the other.

If you want to host in the local country, why don't you do what phranque suggested and go via subdomain option?

If you want folder option, it should resolve straight away to server where the main domain is hosted.

You can geo-target both, subdomains and folder-based solution.

[sdgit]

Yes, subdomain is now looking the better option, as the managed DNS providers say they cannot resolve to directly a folder. The drawback is diminished contribution to root domain authority.

Assuming the subdomain solution, do you think the URL can be masked with the primary domain (i.e. domain.com instead of uk.domain.com) without cross-contaminating search indexing and rank?

[aakk9999]

You have to have subdomain in URL. Otherwise, (unless I am missing something), how it will resolve to subdomain?

Because domain.com will resolve to wherever the domain is hosted, not to subdomain.