Google Will Seek Out HTTPS Pages By Default

[engine]

Google continues to step up its need for HTTPS. Not only is google giving HTTPS a slight ranking boost [webmasterworld.com], it's now going to seek out more HTTPS pages, even if not linked from any page.

....we'd like to announce that we're adjusting our indexing system to look for more HTTPS pages. Specifically, we’ll start crawling HTTPS equivalents of HTTP pages, even when the former are not linked to from any page. When two URLs from the same domain appear to have the same content but are served over different protocol schemes, we’ll typically choose to index the HTTPS URL if:

It doesn’t contain insecure dependencies.

It isn’t blocked from crawling by robots.txt.

It doesn’t redirect users to or through an insecure HTTP page.

It doesn’t have a rel="canonical" link to the HTTP page.

It doesn’t contain a noindex robots meta tag.

The sitemaps lists the HTTPS URL, or doesn’t list the HTTP version of the URL

The server has a valid TLS certificate.
Googlebot Will Seek Out HTTPS Pages By Default [googlewebmastercentral.blogspot.com]

[Leosghost]

You might also try placing a text file in the directory with an ascii art rendition of Alec Guinness saying..
"These are not the HTTPS pages you are looking for.."
or..if you have the RAM..an anigif, so as to get the vague "force manipulating" hand gestures that accompany the phrase..

it was blend27's use of the "far far away" which inspired ;)

This may also work here..
[webmasterworld.com...]

[lucy24]

Imagine if someone [has] a few hundred links to a proper URI on you site but with some curse words in query string. That rule says URI found and moved to a new location, then the linking page gets crawled and you server redirects it.

That's got nothing to do with protocol or hostname. If people are crawling URLs with bogus query strings-- whether it's humans following an error or robots acting out of malice-- it's the path-plus-query that needs to be redirected. The www/https redirect is always the last step; it applies only to requests that have nothing else wrong with them.

[keyplyr]

IMO if it's that important, make HTTPS *the* standard and get rid of the certificates altogether.

[Shaddows]

make HTTPS *the* standard

I'm all for that. As a user, I like SSL, and as an ecom its a necessity.

Other disagree. It can cost money to get a certificate, it adds processing power requirements and multiplies bandwidth.

get rid of the certificates altogether

Can't do that. SSL could intrinsically encrypt all traffic without a cert, but you would not know who you were talking to. It could be a bad guy, decrypting your traffic, reading it, and re-encrypting to forward to intended destination (same process for the response). The certs are supposed to prevent a man-in-the-middle attack such as this.

ETA - Assuming you mean a CA certificate. At base (if I understand these things correctly), a cert is the bit of data you need in order to start an encryption handshake- without a cert, you could not start the encryption process (unless you declared your keys in the clear, which would be self defeating)

[IanKelley]

Now, can anyone explain why google would look for https URLs that are linked from nowhere (their own prose)? I can't figure out who benefits.

It goes with their longer term goal of nudging the web towards secure. Checking both protocols will discover sites which have a valid certificate but aren't using it because none of their links point to https, or are using it but only in a store section. They then favor the https version going forward unless told otherwise = more SSL pages.

Or to put it another way... on most web servers if a domain has a certificate associated with it, any page within the domain can be served via https without any additional configuration. So many sites have the unused ability to serve more of their pages via https.

I personally disagree with Google's goal. I think it's a bit early. The added processor cycles and bandwidth (increasing now that older versions of TLS are being retired) are not insignificant. They will be in the future, even the near future, but not yet. Especially on a small, loaded, server or via a crappy wireless connection (smartphones and tablets, especially in developing areas of the world).

On the other hand much of the web really does need to be more secure and Google does not benefit from this. Right or wrong it seems altruistic.

[ChanandlerBong]

What is the simplest way of "complying" on this?

when this subject did the rounds (in summer? spring?) earlier in the year, I seem to recall that G themselves had got into the certificate business (colour me SHOCKED!)

I did get on board the responsive thing last year and really happy that I did - this one, I'm willing to sit out until I see concrete evidence that our rankings/traffic are suffering. I don't see it as a bad thing per se, but not unusually, G are going overboard with it and requiring https for a flat html info page is overkill.

[EditorialGuy]

I don't see it as a bad thing per se, but not unusually, G are going overboard with it and requiring https for a flat html info page is overkill.

But they aren't "requiring https for a flat html info page." Or for any page at all.

As ogletree pointed out, this is about discovery. Nothing more, nothing less. If you aren't using https, it won't affect you. And if you do switch to https, you should be relieved to know that Google will make it easier for all of your https pages to be discovered and for you to receive full "link juice" on your new https versions of old https pages.

[rainborick]

You don't want to switch to HTTPS for the ranking benefits, which are very small. You want to do this to make your site more secure/trustworthy and to future-proof it. The nytimes article that EditorialGuy referenced earlier in this thread is worth reading.

[IanCP]

You want to do this to make your site more secure/trustworthy and to future-proof it. The nytimes article that EditorialGuy referenced earlier in this thread is worth reading.

I don't agree for the reasons "smallcompany" had already outlined earlier

Hm, in the case of one of my sites it says (Firefox) Untrusted Connection, then asks me if I want to continue. Explorer says Certificate Error, and also offers to continue.

I consider it as absolutely absurd a site about painting widgets blue would remotely require SSL. Going to such sites and receiving alerts is down right annoying, almost as bad as continual messages about Adobe Flash.

Webmasters are becoming their own worst enemies. Pages riddled with masses of Javascript, advertisements, tracking, and now https with potentially dodgy certificates.

I really would like someone to concisely explain to me why your average information site requires https, and I don't think the principles set out in the New York Times blog post titled "Embracing HTTPS" are really applicable to "Mary's Cake Recipe" site. Or my old lectures on electronics for that matter.

[blend27]

And so there You go: FACTS

I took one of the sub-domains offline last April(2015) completely due to someones hacked site on shared server got everybody infected.

In May I moved all sub.domains and www version to a new hosting provider.

The only thing I have uploaded to this particular sub.domain was a

1. .HTACCESS file

RewriteEngine on
RewriteCond %{HTTP_HOST} ^sub.domain.com$ [NC]
RewriteRule .* - [F] 

2. robots.txt(blank)
3. and 404(all requests were rewritten to it) page so that would give me access to headers when someone is trying to access the site.

Re-launched the sub.domain.com 3 days ago.

Today using this site:sub.domain.com

At position 2.

https ://sub.domain.com/
A description for this result is not available because of this site's robots.txt – learn more. - [support.google.com...]

1. robots.txt(blank) always.
2. If one tries to access it via https, one would get untrusted connection message.
3. By adding exception, at least in FF You'd still get blank robots.txt(200)

Nothing was ever blocked by robots.txt.. Home page was returning 403- Directory listing denied and Every page was returning 404 because the site just was not there, never mind HTTPS.

The main site, including about 10 sub-domains, have been online since 2001.

This particular sub.domain 22 pages that describe how to plant tomatoes and horseradish in the Northern NJ, USA at Spring time for God's sake. The pages were one for each of 20 weeks starting March with the report including pictures.

No Ecommerce, no Ads, no banners, no link-outs, no tracking by third party sources, same exact WHOIS/OWNER since the domain was registered, same DNS until May, no contact form(just email address image).

The traffic would spike in Feb to a whooping 250-300 visitors per month MAX and would die down in May to almost 0 according to IIS log parser.

How much confusion this might be to the actual user when they search for a domain in SERP(and a lot of people do)?

Never had SSL on it. And never will.

[EditorialGuy]

Never had SSL on it. And never will.

OK, so you don't care that Google is making life easier for people who are migrating to https. More to the point, Google's announcement doesn't affect you, so why let it bother you?

[lucy24]

https with potentially dodgy certificates

I think the warning message means that there's no certificate at all, and the browser is alerting you to this fact before it takes the step of requesting the page. (Query: How would encrypted content get handled if there's no certificate explaining what to do with it?)

I'm trying to think of an analogy and all I can come up with is a security guard asking to see your credentials for entering a building before they check whether the person you want to see is there right now, or works there at all, or possibly whether the firm they ostensibly work for even has offices in the building.

Does "rankings boost" mean that your competitor's gratuitously-https site might gain a couple of spots, thereby coming in ahead of your own purely-http site? Seems like that would affect everyone.

[blend27]

OK, so you don't care that Google is making life easier for people who are migrating to https. More to the point, Google's announcement doesn't affect you, so why let it bother you?

priceless.

[raseone]

https with potentially dodgy certificates

He's actually kinda right. When you see those horrifying warning messages it is because the host is providing a "self signed certificate". It will still do its job but it puts the user in the impossible position of deciding if the y "trust" a website that they may have never visited before.

A "self signed certificate" is not issued by a trusted "certificate authority" so the browser does not "trust" it by default. You can tell your browser to trust certificates from that site & the warning will never appear again however the warning messages are more than enough to scare away the average, layman web user.

[incrediBILL]

Now that HTTPS certs are free, the whole issue is moot.

Get a FREE cert, be compliant.