Forum Moderators: Robert Charlton & goodroi
64.233.183.107
64.233.187.99
66.249.85.107
If you do a site: search using Google UK's 'UK results only', the index pages of ten .com domains and three .net domains (out of 65 that I manage) are not listed.
1) All sites hosted in the UK across 4 different servers in 3 different DCs
2) No link exchanges, very little outward linking, no excessive inward linking. No inter-linking. These are not directories, MFA, affiliate. They are contact/services offered websites for UK companies and sole traders. Some are dynamic, some static. Size from under ten to under one hundred pages.
3) No canonical issues, no dupe content issues, no over-optimisation. I use the same techniques and links for all my sites. Number of affected sites has not grown since this problem was spotted. Unaffected sites re-cached this week.
4) Internal pages all listed AND RANKING for their terms.
Would other UK based webmasters like to share their experiences? Perhaps we can collectively contact Google and refer them to this thread.
i think a few people on this thread (it's been running since august!) had tried the same with decent results so i gave it a whirl.
Mine lists on a competitive "blue widgets" keyword no.2 in UK/web normally on google.co.uk and it was listing on "search the web" but not in UK.
(UK server, UK IP, .com Domain)
I didn't do too much to the content, mainly a basic re-jig, but i did add about 10 more location keywords into the body and title(i.e. UK, england, wales Britain etc.)
The site was crawled that day and the new version was in the index and stayed put within 36 hours.
It might have been a coincidence, but it's strange that this has worked a few times now.
Cheers,
hughie
One of my longest affected sites has "brighton" in the domain name and is optimised for "brighton", "west sussex", "uk". Others have always been optimised for "london", "kent" etc etc.
As far as changing content goes - two other affected sites dynamically update content every visit to the page.
Tweaking won't change anything. But it's always good to look at your site and ask yourself if it could be better though :)
This topic has received quite a bit of attention in the Google Webmaster Discussion Forum over the past week.
Funnily enough, if you browse through the topics form the past week, it seems that Google has removed the threads about the topic, but you can still access the disccusion directly if you have the URLs.
Needless to say, no one from Google has said a word about it.
Anyway, from the discussion, I have about 10 sites from both Australia and UK (including mine) that I shall be watching to see if they come back.
I am also now quite positive it is a canonical type issue as all the sites affected have links on their site to http://www.example.com/index.html or something like that.
Google picks it up, but now they are indexing http://www.example.com/home.php (a PPC landing page only linked to in Google ads and formerly in my sitemap.xml file) in the AU index. Unfortunately home.php has the same <head> as www.example.com, thus Google treating them the same.
Anyway, I have significantly changed the h1 tag on www.example.com and added home.php to robots.txt.
This is becoming very frustrating.
I don't think it is a canonical issue: my own site uses www.mysite.com throughout (no instances of mysite.com or www.mysite.com/index...), and has come and gone several times.
Let us hope Google now have it fixed (...he said, with no great conviction).
I don't think it is a canonical issue: my own site uses www.mysite.com throughout (no instances of mysite.com or www.mysite.com/index...), and has come and gone several times.
yes, but have you linked to index.... within the past year? For my site, when "Pages from Australia" is selected, it returns a heap of previous supplemental results which haven't been redirected or 404'ed - eg http://www.example.com/blah/?id=xx. as opposed to http://www.example.com/blah/ which is the current link.
Without "Pages From Australia" the site command returns the proper results.
We didn't do anything to make this happen, it just came back after 3-4 weeks of not appearing.
I think this is something buggy at googles end and just being patient will fix the problem for most people.
I don't have (and haven't had) any problems with supplemental results. Generally, site: searches on the UK DC's have correctly returned all my current pages (other than the homepage) since August. Specifically the problem has been exclusion of the homepage (and only the homepage) from UK-only searches.
Another thing i've noticed this past few days, for certain keywords sites are not showing up in the SERPS, only for certain keywords, but when you add "&filter=0" to the end of the search string, they show up - i remember seeing this a lot a few months back.
My site is a .net and I am seeing .coms and .nets affected the same way in my SERP
I am getting a bit sick of this and am contemplating 301 redirect to a .co.uk domain but am really scared of risking it! Anyone taken drastic measures like this?
[edited spelling]
[edited by: Jez123 at 9:25 am (utc) on Nov. 8, 2006]
My site is still affected, and I still believe this is a canonical/duplicate content issue.
My theory is that Google has introduced a new geotargeting filter which when duplicate content within the same site is discovered, it tries to decide which is more country specific (eg "More Australian" or "More British") based on inbound links, amongst other factors.
This duplicate content may come from the supplemental index, or may have been included in your Google sitemap, it doesn't matter - Google will still find it and index it in the country specific index, bumping your home page/other pages out of the index.
What this means is that if you have ever (past year maybe?) linked, or included in your Google sitemap a reference to www.example.com/index.htm, [example.com,...] www.example.com/?blah=blah etc etc, it will still be in Google's index and will be ripe for indexing and inclusion in your country's specific index.
This means that you have to totally and utterly annihilate any canonical issues that you have, have had or will have in the future.
Things I have done in the past week/few days:
Redirecting non-www to www (or vice versa). If you have a DNS wildcard entry which goes to your site, you need to set up a redirect for that as well.
If a page has unneeded query parameters, do a redirect to the proper page.
Redirect https:// to http://
Disallow similar pages to my home page in robots.txt (eg PPC landing pages).
I base this theory on random searches, URLs given in Google's Webmaster Discussion Group and my own site. Hopefully I am right and my site will be back soon.
Google totally ignored my home page on that occasion in the same way that it did with this "uk results" issue.
However that doesn't explain why it appears perfectly well in the "web" results.
I think a 301 to a new .co.uk domain would be the Google ranking equivilent of suicide.
I agree. Since BigDaddy cam along you will effectively sandbox your site if you move domains - doesn't matter whether you re-brand to a new domain or just switch extension. I've had two sites hit by this; nice of Google to tell us that 301s were no longer honoured wasn't it?
Re : Canonical Issues
I have had a few sites affected where on top of *never* linking internally to /index.htm or similar (I've not done this on a site for 3 years I always use the domain - in fact I always use absolute links, never relative for all pages on all sites) I use htaccess to avoid this ever happening.
It's a reasonable explanation but it just doesn't fit what I'm seeing on more than one site.
I have had a few sites affected where on top of *never* linking internally to /index.htm or similar (I've not done this on a site for 3 years I always use the domain - in fact I always use absolute links, never relative for all pages on all sites) I use htaccess to avoid this ever happening.It's a reasonable explanation but it just doesn't fit what I'm seeing on more than one site.
Remember it's not just a canonical issue, it is a duplicate content issue too. That means if, for example, you have had a page with the same <head> as your home page, then it could be bumping your home page out of your country's index.
I've seen this on my own site and when I do a search on google.com.au for the name of a large Australian company who runs many sites with TLDs - one site has duplicate titles and meta descriptions and is affected by this problem. All other sites have a unique <head> and aren't affected by this problem.
It's a reasonable explanation but it just doesn't fit what I'm seeing on more than one site.
Me too. I am not seeing this at all.
Clearly it isn't over yet.
if, for example, you have had a page with the same <head> as your home page
M_Bison, again this just isn't what I'm seeing on the sites I have affected. Some only have 10 pages or less and the titles, descriptions and keywords are unique for all.
I have posted on the Google Webaster forums linking to this thread and one of W3bw0rks0p; I've also posted on Matt Cutt's blog (again, he did look at my last post but said there wasn't an issue) and I've even contacted our Google Adwords rep about this to see if anyone from Google will take an interest.
I see another site in my SERP that has recovered after being affected for about a month or more - I don't think this is over for them though. This is another one of those tricky little google puzzles that we may never know the reason why.
I am a .net hosted in the UK aimed at UK clients but with inbound links from throughout the world.
My view - don't worry: it's almost certainly nothing you've done. It's them! It's got to be.....hasn't it?
Now where was that darkened room?
Only two changes have been made since my last post on 28 Oct;
1. Update our new product data pages, which is a routine job for every 4 weeks, so I don't think this is the reason.
2 Change one internal link from mysite.com/index.html to mysite.com/
I don't know is this helped us back to UK search or not, but according to other topics in this forum about duplicate content and one URL per one page issues, it might make little sense. Or, simply it was Google bug?
Anyway, I just hope that our site not going to disappear again from UK SERP, and wish those who still out in the cool back to the UK SERP soon, good luck.
Like others our main site has been greatly affected by this issue, but since yesterday evening things seem back to normal.
The actions we took, which may or may not have contributed to this, are as follows:-
1. Notified Google that we wanted their indexing to show www.example.com rather than example.com
2. Replaced all links to http://www.example.com/index.html on our website with http://www.example.com/
3. Replaced all links to http://www.example.com/index.html in Google Adwords with http://www.example.com/
The one thing we haven't done yet is a 301 redirect from http://example.com/ to http://www.example.com/
However, if we have similar probs in future it's top of my list of actions to try.
Hopefully others will find the steps we've taken work from them as well.
[edited by: tedster at 9:19 pm (utc) on Nov. 25, 2006]
[edit reason] use example.com [/edit]
this was a google glitch which seems to have been fixed, until the next time.
GingerTom1403 said - "The actions we took, which may or may not have contributed to this"
My input - I don't think that your actions had any effect, nor contributed to your return to the index!
UK_Web_Guy said, - "this was a google glitch which seems to have been fixed, until the next time."
My input - it was, is still, and will continue to be something wrong at the Google end!
Has anyone actually made any progress in getting one of the Google reps around here look into this? It surely can't be something that they "want" to be happenning, it just wouldn't make any sense.
Simon.
I don't like it when it happens, but to be honest I don't know that it makes a huge difference to traffic, not many people click on pages from the UK, most just type in the query and hit enter.
Interested to hear if anybody thinks differently on this...
