SEO Rankings- secure pages and 301 redirects

[Ajay_Stroll]

Anyone know if MSN and Yahoo will spider a secure page (https://)?

Do you know of any ranking penalties associated with https OR 301 redirects TO an https?

An answer addressing BOTH MSN and Yahoo would be greatly appreciated. Thanks!

[pageoneresults]

Anyone know if MSN and Yahoo will spider a secure page (https://)

Yes they will.

The question now becomes why? There are additional processes involved with https vs http. There are many other challenges in promoting https links, they are not natural for one.

We typically recommend that you serve a robots.txt file for all https that Disallows the entire site. You really don't want anything that requires secure protocol getting indexed. At least that is how we look at it. Login pages serve no purpose in the equation. And neither do most other pages that qualify for https.

Talk to us. Why are you redirecting to https?

[jwolthuis]

We typically recommend that you serve a robots.txt file for all https that Disallows the entire site.

Are you saying that [mysite.com...] would be the "normal" file, but [mysite.com...] would be a different file, and would Disallow every page?

How would a bot know to retrieve the https version of robots.txt?

Since the contents of the robots.txt file makes no distinction between the protocol used, doesn't it seem rather dangerous to offer a version of robots.txt that makes your entire site off limits?

[Ajay_Stroll]

Our site -- which has been indexed in the engines for many years -- is mainly made up of pages which drive purchase transactions. We're an ecommerce site. We have recently redesigned our shopping cart/purchase checkout process such that pages which rank in the engines and which used to be http (non-secure) pages now are https (secure) pages as they now include a one-step check-out. We're worried that we might get unfairly penalized for the change, even though the content of the pages is the same and just as relevant as it was in the past. The only change is the inclusion of the one-step checkout which has forced to make them https pages.

Do you think that we should be concerned to lose our rankings in Yahoo! or MSN or, for that matter, in Google in having made the switch to https?

These are the pages to which we're re-directing to (from our homepage) using the 301 mechanism.

[particleman]

Seems google at least can make the distinction between different robots.txt files. I followed a guide online using rewrite rules to serve a different robots.txt for my SSL pages. Since my site was fully accessible from [,...] somehow over the years google had indexed huge numbers of products under our secure site. We took several steps to isolate where googlebot even got on the secure site, but the SSL robots.txt seemed to start getting pages redirected back to http in a week or so.

However, timing wise the canonical tag was agreed on shortly after this, this was the ultimate solution to the problem since the tag allows you to use an absolute URL, we loaded that in and now ALL the products are getting moved over to [....] My suggestion, use the new canonical tag as google says it will work as a redirect, plus protect you from duplicate content issues.

[pageoneresults]

How would a bot know to retrieve the https version of robots.txt?

Through a little bit of scripting that looks like this for us folks on Windows...

RewriteCond %HTTPS ^on$
RewriteRule /robots.txt /robots.https.txt [I,O,L]

That's about as far as I can take it. I only know enough to cut and paste and double check to be sure everything is doing what it should be doing. And it seems to be working just fine. ;)