Welcome to WebmasterWorld Guest from 54.227.125.200

Message Too Old, No Replies

"UK results only" - site: command shows index page missing

A problem in Google UK (UK results only filter) affects non .co.uk domains

     

fishfinger

8:03 am on Aug 18, 2006 (gmt 0)

10+ Year Member



Except on these UK datacentres

64.233.183.107
64.233.187.99
66.249.85.107

If you do a site: search using Google UK's 'UK results only', the index pages of ten .com domains and three .net domains (out of 65 that I manage) are not listed.

1) All sites hosted in the UK across 4 different servers in 3 different DCs
2) No link exchanges, very little outward linking, no excessive inward linking. No inter-linking. These are not directories, MFA, affiliate. They are contact/services offered websites for UK companies and sole traders. Some are dynamic, some static. Size from under ten to under one hundred pages.
3) No canonical issues, no dupe content issues, no over-optimisation. I use the same techniques and links for all my sites. Number of affected sites has not grown since this problem was spotted. Unaffected sites re-cached this week.
4) Internal pages all listed AND RANKING for their terms.

Would other UK based webmasters like to share their experiences? Perhaps we can collectively contact Google and refer them to this thread.

hughie

1:30 pm on Oct 30, 2006 (gmt 0)

10+ Year Member



Hi Wilburforce

i think a few people on this thread (it's been running since august!) had tried the same with decent results so i gave it a whirl.

Mine lists on a competitive "blue widgets" keyword no.2 in UK/web normally on google.co.uk and it was listing on "search the web" but not in UK.
(UK server, UK IP, .com Domain)

I didn't do too much to the content, mainly a basic re-jig, but i did add about 10 more location keywords into the body and title(i.e. UK, england, wales Britain etc.)

The site was crawled that day and the new version was in the index and stayed put within 36 hours.

It might have been a coincidence, but it's strange that this has worked a few times now.

Cheers,
hughie

fishfinger

10:46 am on Oct 31, 2006 (gmt 0)

10+ Year Member



My view is that it is just coincidence.

One of my longest affected sites has "brighton" in the domain name and is optimised for "brighton", "west sussex", "uk". Others have always been optimised for "london", "kent" etc etc.

As far as changing content goes - two other affected sites dynamically update content every visit to the page.

Tweaking won't change anything. But it's always good to look at your site and ask yourself if it could be better though :)

Wibfision

9:33 pm on Nov 4, 2006 (gmt 0)

10+ Year Member



Hi Guys, My homepage is now back within the UK results. I didn't change anything. :-)

M_Bison

5:43 am on Nov 5, 2006 (gmt 0)

5+ Year Member



That's good to hear webfision.

This topic has received quite a bit of attention in the Google Webmaster Discussion Forum over the past week.

Funnily enough, if you browse through the topics form the past week, it seems that Google has removed the threads about the topic, but you can still access the disccusion directly if you have the URLs.

Needless to say, no one from Google has said a word about it.

Anyway, from the discussion, I have about 10 sites from both Australia and UK (including mine) that I shall be watching to see if they come back.

I am also now quite positive it is a canonical type issue as all the sites affected have links on their site to http://www.example.com/index.html or something like that.

M_Bison

10:31 pm on Nov 5, 2006 (gmt 0)

5+ Year Member



OK, so I update [example.com...] to 301 to http://www.example.com.

Google picks it up, but now they are indexing http://www.example.com/home.php (a PPC landing page only linked to in Google ads and formerly in my sitemap.xml file) in the AU index. Unfortunately home.php has the same <head> as www.example.com, thus Google treating them the same.

Anyway, I have significantly changed the h1 tag on www.example.com and added home.php to robots.txt.

This is becoming very frustrating.

Wilburforce

11:08 am on Nov 6, 2006 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month



My homepage is also back this morning (#1 for primary search term), having been out since 3 October. Last crawled on 28 October. No changes at this end since previous crawl.

I don't think it is a canonical issue: my own site uses www.mysite.com throughout (no instances of mysite.com or www.mysite.com/index...), and has come and gone several times.

Let us hope Google now have it fixed (...he said, with no great conviction).

M_Bison

11:15 am on Nov 6, 2006 (gmt 0)

5+ Year Member



I don't think it is a canonical issue: my own site uses www.mysite.com throughout (no instances of mysite.com or www.mysite.com/index...), and has come and gone several times.

yes, but have you linked to index.... within the past year? For my site, when "Pages from Australia" is selected, it returns a heap of previous supplemental results which haven't been redirected or 404'ed - eg http://www.example.com/blah/?id=xx. as opposed to http://www.example.com/blah/ which is the current link.

Without "Pages From Australia" the site command returns the proper results.

briggidere

11:33 am on Nov 6, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



our site has come back into the "pages from the UK" results now. it happend about 10 days ago.

We didn't do anything to make this happen, it just came back after 3-4 weeks of not appearing.

I think this is something buggy at googles end and just being patient will fix the problem for most people.

Wilburforce

12:14 pm on Nov 6, 2006 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month



A couple of instances in the last year, yes. I cleaned it up in August, when the problem first affected me, although I don't think this has made any significant difference. Canonical issues wouldn't have a geo-specific effect.

I don't have (and haven't had) any problems with supplemental results. Generally, site: searches on the UK DC's have correctly returned all my current pages (other than the homepage) since August. Specifically the problem has been exclusion of the homepage (and only the homepage) from UK-only searches.

UK_Web_Guy

8:56 am on Nov 7, 2006 (gmt 0)

10+ Year Member



Still seeing this issue coming and going for some sites - some have been affected by it for a long time now.

Another thing i've noticed this past few days, for certain keywords sites are not showing up in the SERPS, only for certain keywords, but when you add "&filter=0" to the end of the search string, they show up - i remember seeing this a lot a few months back.

Jez123

9:24 am on Nov 8, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



My site has been affected again! I will change the titles of the affected pages again as this caused (or seemed to) the index to come back last two times. I am noticing that it isn't just the main index page that can be affected by this - I have a few mini sites within the same site that have been hit as well. Not all are the classic index.html but can be descriptive-product-index.html as well.

My site is a .net and I am seeing .coms and .nets affected the same way in my SERP

I am getting a bit sick of this and am contemplating 301 redirect to a .co.uk domain but am really scared of risking it! Anyone taken drastic measures like this?

[edited spelling]

[edited by: Jez123 at 9:25 am (utc) on Nov. 8, 2006]

M_Bison

10:05 am on Nov 8, 2006 (gmt 0)

5+ Year Member



I think a 301 to a new .co.uk domain would be the Google ranking equivilent of suicide.

My site is still affected, and I still believe this is a canonical/duplicate content issue.

My theory is that Google has introduced a new geotargeting filter which when duplicate content within the same site is discovered, it tries to decide which is more country specific (eg "More Australian" or "More British") based on inbound links, amongst other factors.

This duplicate content may come from the supplemental index, or may have been included in your Google sitemap, it doesn't matter - Google will still find it and index it in the country specific index, bumping your home page/other pages out of the index.

What this means is that if you have ever (past year maybe?) linked, or included in your Google sitemap a reference to www.example.com/index.htm, [example.com,...] www.example.com/?blah=blah etc etc, it will still be in Google's index and will be ripe for indexing and inclusion in your country's specific index.

This means that you have to totally and utterly annihilate any canonical issues that you have, have had or will have in the future.

Things I have done in the past week/few days:

Redirecting non-www to www (or vice versa). If you have a DNS wildcard entry which goes to your site, you need to set up a redirect for that as well.
If a page has unneeded query parameters, do a redirect to the proper page.
Redirect https:// to http://
Disallow similar pages to my home page in robots.txt (eg PPC landing pages).

I base this theory on random searches, URLs given in Google's Webmaster Discussion Group and my own site. Hopefully I am right and my site will be back soon.

hughie

11:38 am on Nov 8, 2006 (gmt 0)

10+ Year Member



i'm with you on the dupe content issue, whilst my site is still fine (fingers crossed) the effects were identical to a previous issue when some shark lifted the entire front page of my website.

Google totally ignored my home page on that occasion in the same way that it did with this "uk results" issue.

However that doesn't explain why it appears perfectly well in the "web" results.

fishfinger

1:32 pm on Nov 8, 2006 (gmt 0)

10+ Year Member



I think a 301 to a new .co.uk domain would be the Google ranking equivilent of suicide.

I agree. Since BigDaddy cam along you will effectively sandbox your site if you move domains - doesn't matter whether you re-brand to a new domain or just switch extension. I've had two sites hit by this; nice of Google to tell us that 301s were no longer honoured wasn't it?

Re : Canonical Issues

I have had a few sites affected where on top of *never* linking internally to /index.htm or similar (I've not done this on a site for 3 years I always use the domain - in fact I always use absolute links, never relative for all pages on all sites) I use htaccess to avoid this ever happening.

It's a reasonable explanation but it just doesn't fit what I'm seeing on more than one site.

M_Bison

9:52 pm on Nov 8, 2006 (gmt 0)

5+ Year Member



My site is back, as well as another site mentioned on the Google's discussion forum. Hopefully it stays that way.

M_Bison

10:13 pm on Nov 8, 2006 (gmt 0)

5+ Year Member



I have had a few sites affected where on top of *never* linking internally to /index.htm or similar (I've not done this on a site for 3 years I always use the domain - in fact I always use absolute links, never relative for all pages on all sites) I use htaccess to avoid this ever happening.

It's a reasonable explanation but it just doesn't fit what I'm seeing on more than one site.

Remember it's not just a canonical issue, it is a duplicate content issue too. That means if, for example, you have had a page with the same <head> as your home page, then it could be bumping your home page out of your country's index.

I've seen this on my own site and when I do a search on google.com.au for the name of a large Australian company who runs many sites with TLDs - one site has duplicate titles and meta descriptions and is affected by this problem. All other sites have a unique <head> and aren't affected by this problem.

Jez123

12:52 pm on Nov 9, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



It's a reasonable explanation but it just doesn't fit what I'm seeing on more than one site.

Me too. I am not seeing this at all.

Wilburforce

8:52 am on Nov 10, 2006 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month



UK & global #2 for key search term in my sector has gone missing in the UK again this morning. Info: finds the homepage but site: doesn't (i.e. the original problem conditions apply).

Clearly it isn't over yet.

fishfinger

9:33 am on Nov 10, 2006 (gmt 0)

10+ Year Member



if, for example, you have had a page with the same <head> as your home page

M_Bison, again this just isn't what I'm seeing on the sites I have affected. Some only have 10 pages or less and the titles, descriptions and keywords are unique for all.

I have posted on the Google Webaster forums linking to this thread and one of W3bw0rks0p; I've also posted on Matt Cutt's blog (again, he did look at my last post but said there wasn't an issue) and I've even contacted our Google Adwords rep about this to see if anyone from Google will take an interest.

asher02

2:08 pm on Nov 10, 2006 (gmt 0)

5+ Year Member



My site is is also effected as from today. A com site hosted in the US is no longer showing for pages from US. Only the home page is effected as other pages are in the index. Does anybody else with US hosting have the same problem when searching for US pages?

Jez123

9:46 am on Nov 14, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Well, my usual trick of simply changing the index page title didn't work this time! 3rd time unlucky. Looks like I am going to suffer for a while.

I see another site in my SERP that has recovered after being affected for about a month or more - I don't think this is over for them though. This is another one of those tricky little google puzzles that we may never know the reason why.

lee_sufc

2:48 pm on Nov 17, 2006 (gmt 0)

10+ Year Member



My site was affected by this about 1.5 months ago and came back after about 3 weeks.. Having hoping that was the end...the problem is back again!

I am a .net hosted in the UK aimed at UK clients but with inbound links from throughout the world.

mcdermc

3:35 pm on Nov 17, 2006 (gmt 0)

10+ Year Member



One of my sites just returned yesterday to the uk results index (.com UK hosted) for the 2nd time since August. In both cases it was 'disappeared' for a whole month each time.

My view - don't worry: it's almost certainly nothing you've done. It's them! It's got to be.....hasn't it?

Now where was that darkened room?

kmick

11:25 am on Nov 21, 2006 (gmt 0)

10+ Year Member



After 3 months missing from the UK search, our home page is back. Two of our main 2 words phrase search is now #1 and #2 in UK SERP.

Only two changes have been made since my last post on 28 Oct;

1. Update our new product data pages, which is a routine job for every 4 weeks, so I don't think this is the reason.

2 Change one internal link from mysite.com/index.html to mysite.com/

I don't know is this helped us back to UK search or not, but according to other topics in this forum about duplicate content and one URL per one page issues, it might make little sense. Or, simply it was Google bug?

Anyway, I just hope that our site not going to disappear again from UK SERP, and wish those who still out in the cool back to the UK SERP soon, good luck.

GingerTom1403

1:47 pm on Nov 25, 2006 (gmt 0)

5+ Year Member



www homepage back in SERPS and site:www.domainname.com query results

Like others our main site has been greatly affected by this issue, but since yesterday evening things seem back to normal.

The actions we took, which may or may not have contributed to this, are as follows:-

1. Notified Google that we wanted their indexing to show www.example.com rather than example.com

2. Replaced all links to http://www.example.com/index.html on our website with http://www.example.com/

3. Replaced all links to http://www.example.com/index.html in Google Adwords with http://www.example.com/

The one thing we haven't done yet is a 301 redirect from http://example.com/ to http://www.example.com/

However, if we have similar probs in future it's top of my list of actions to try.

Hopefully others will find the steps we've taken work from them as well.

[edited by: tedster at 9:19 pm (utc) on Nov. 25, 2006]
[edit reason] use example.com [/edit]

UK_Web_Guy

3:04 pm on Nov 25, 2006 (gmt 0)

10+ Year Member



loads of sites back showing index pages on .co.uk site: searches.

this was a google glitch which seems to have been fixed, until the next time.

lee_sufc

3:12 pm on Nov 25, 2006 (gmt 0)

10+ Year Member



not fixed for us yet

mcdermc

3:44 pm on Nov 25, 2006 (gmt 0)

10+ Year Member



My UK hosted dot com index page just disappeared again from the uk filtered index today for the 3rd time of asking...!

GingerTom1403 said - "The actions we took, which may or may not have contributed to this"
My input - I don't think that your actions had any effect, nor contributed to your return to the index!

UK_Web_Guy said, - "this was a google glitch which seems to have been fixed, until the next time."
My input - it was, is still, and will continue to be something wrong at the Google end!

swones

10:06 am on Nov 27, 2006 (gmt 0)

10+ Year Member



Not back here yet either, the first time it happened I tried the suggestions that others made on this thread and it came back for about 2-3 weeks. Now it's been gone again for as long.

Has anyone actually made any progress in getting one of the Google reps around here look into this? It surely can't be something that they "want" to be happenning, it just wouldn't make any sense.

Simon.

UK_Web_Guy

10:49 am on Nov 27, 2006 (gmt 0)

10+ Year Member



Sorry to hear not all have come back, some sites which have been missing for ages came back in this latest refresh.

I don't like it when it happens, but to be honest I don't know that it makes a huge difference to traffic, not many people click on pages from the UK, most just type in the query and hit enter.

Interested to hear if anybody thinks differently on this...

This 211 message thread spans 8 pages: 211
 

Featured Threads

Hot Threads This Week

Hot Threads This Month