google canonical problems with http and https with new update

google replaced http homepage with https homepage

[dont_get_it]

Just noticed after seeing that our site dropped a number of positions on terms that we've consistently positioned well on, that when I query site:www.my_site.com that Google has replaced the indexed version of www.my_site.com with [my_site.com....]

Despite thousands of incoming links to [my_site.com,...] I see that one of our partners has linked to us via [my_site.com....] All links within the website that direct the user "home" point to an absolute [my_site.com....]

How could one new incoming link cause Google to replace the non-secure version of the page with the secure version of the same page?

Any advise on how to remedy? I attempted a 301 redirect using Apache Mod Rewrite and the rule:

RewriteRule [my_site.com...] [my_site.com...] [L,R=301]

However it doesn't appear to be doing the trick.

Any and all advise on how to notify Google of the problem (is this even possible) and/or how to remedy via a 301 so that the problem works itself out over time is greatly appreciated.

[tedster]

Another report of the same issue came in today, so I'm adding it below

[dont_get_it]

I noticed today that when searching for a specific term on Google, that both the secure and non-secure URL to our site's index page were listed within the results. After seeing this, I used the site: command and noticed that only the secure version of our page is displayed within results.

Any advise on how to remedy this? I would hate to get hit with duplicate content penalities.

By the way, every link to our home page within the site uses an absolute URL to point to the home page, and practically all of the thousands of inbound links to our site point to the non-secure (I did find a few instances of some clients and partners linking to the secure address).

Thanks for any help or suggestions!

[ansible]

I'm having the same problem. I tried playing with the robots.txt tool in google sitemaps, but it doesn't look like doing a Disallow for https:// will work because it treats https:// as another domain all together.

I'm scared of putting a Disallow for all, in my https directory because it might remove all of my site's content.

Anyone know how to remove only https:// content via robots.txt?

[dont_get_it]

Someone was kind enough to share some rewrite rules for Apache that will at least alert the spiders with a 301 redirect for the secure URi. I've placed on the site and verified that the correct headers are being returned. Of course, in order to use this you'll need to verify that the Mod Rewrite module is available on your Apache server.

RewriteCond %{SERVER_PORT} ^443
RewriteCond %{REQUEST_URI}!^/cgi-bin/.*
RewriteCond %{REQUEST_URI}!^/images/.*
RewriteCond %{REQUEST_URI}!^/includes/.*
RewriteRule ^(.*)$ [my_site.com...] [R=301,L,QSA]

This will direct any and all requests to URi's not within the excluded directories (in this example /cgi-bin/, /images/, and /includes/ would be exempted from the rewrite rule since requests to these directories DO need to be accessed securely) accessed via https:// to the http:// version of the same URi. The server will return a 301 redirect to alert any visiting spiders/bots.

Works great for me... Now I just have to wait and see how quickly the Google bot picks up on it and how long it takes to get the secure reference out of the index.

[tedster]

Using the removal tool in this sutuation has made a LOT of trouble for some sites, with both their secure and regular urls disappearing. The best practice is, from the start, to set up the secure cert on a subdomain, like secure.example.com. If you do that, then the regular pages cannot resolve with https. However, if you;ve already got troubles...

Google suggests serving two robots.txt files:

Each port must have its own robots.txt file. In particular, if you serve content via both http and https, you'll need a separate robots.txt file for each of these protocols. For example, to allow Googlebot to index all http pages but no https pages, you'd use the robots.txt files below.

For your http protocol (http://yourserver.com/robots.txt):
User-agent: *
Allow: /

For the https protocol (https://yourserver.com/robots.txt):
User-agent: *
Disallow: /

[google.com...]

[dont_get_it]

Tedster,

Thanks for the info.

Regarding the two robots.txt files, this would only work if you actually have two seperate document directories, correct?

In our case, we have one document directory on the server that is serving both secure and non-secure content, with the benefit being that none of the content has to be duplicated. Unless the robots recognized something like robots.txt and secure-robots.txt, I can't see how this would be implmented into our set up.

[arnarn]

I'm tending to agree that G has an issue related to https:// and their "fixing" canonical problems.

Maybe this is something they'll have to go back in and "repair the damages" like they have in other cases.

I've seen a suggestion in another forum to add the following to your header tag:

<?
// Print out suitable robots meta tag depending on the connection mode

if (isset($_SERVER['HTTPS']) && strtolower($_SERVER['HTTPS']) == 'on')
{
echo '<meta name="robots" content="noindex,nofollow">';
}
else {
echo '<meta name="robots" content="index,follow">';
}
?>

Would the above be a workable solution?

[arnarn]

A bump and a twist on this topic:

Being concerned about [ourdomain.com...] showing up as #1 on a site:ourdomain.com listing, we implemented the change to our index.htm file to change the meta robots to "noindex,nofollow" when our site is accessed via [ourdomain.com...] and to serve up meta robots to "index,follow" for [ourdomain.com...] access.

I am concerned that somehow G might treat https and http as the SAME when indexing and somehow get confused and apply the "noindex,nofollow" to [ourdomain.com....]

Also, it looks like overnight, most of our site:ourdomain.com results are coming out with just ourdomain.com/ as the #1 entry (no http: or https:)

Should we keep what we put in place our changes to deal with [ourdomain.com...] differently than [ourdomain.com,...] or if it looks like G is fixing the https:// issue, should we back out of our changes?

[Ellio]

I am something of an authority on this subject as we had excactly the same problem with Google listing https versions of pages with no links to them and bumping out the correct http pages.

We cured the problem with Googles help.

1. Changed ALL links to absolute.

2. Made sure that https pages were now served on a subdomain or seperate domain.

3. Made sure that all indexed https pages returned 404 errors.

4. Asked Google to delete all indexed https pages. They did this but only when they showed true windows 404 errors. Custom 404 error pages are not good enough for G.

5. Waited for Google to re-index http pages within a few days.

Result was all pages back and ranking as before with the added protection that absolute links provide.

Hope this helps.

[benc007]

arnarn,

Great post!

[g1smd]

I waited months for a publisher to update a PDF document with revised information, and then many more months for Google to reindex that new content.

Last week, I noticed that it had been reindexed, and yes it is now listed twice, once as http: and the other as https: - I thought nothing of it at the time (it isn't my site, all I wanted was for the new information to be reindexed and the old information to be dumped), but having seen another example today, maybe this problem is already more widespread than many imagine.

Yet another Google bug that has been uncovered in the last few weeks. How many more are there?

.

[google.com...]

[edited by: g1smd at 6:26 pm (utc) on May 10, 2006]

[dont_get_it]

Hey,

I wanted to give everyone an update on what happened in our case. We sent a request to Google regarding the problem, and implemented a 301 redirect using the info in a previous post. We ended up getting a canned response from Google, however it appears that the problem has been resolved and we're back up to our normal (actually, better than normal) position.

Don't know if it was the 301, someone at Google actually changing something before sending a canned response, or a combination of the two, but it fortunately worked out for us.

Good luck to others with the same problem!

[arnarn]

they're BACK!

The [ourdomain.com...] showing up #1 on a site:ourdomain.com.

The problem appeared about 2 weeks ago, disappeared about a week ago and now is pretty much showing up across the board on all DCs (mcdar).

It looks like our changing [ourdomain.com...] to return a meta "robots" as "noindix,nofollow" doesn't make a difference.

Does anybody have a similar finding?