RewriteRule ^ https://www.example.com/$1 [L,R=301]
This would never have worked as intended (regardless of server upgrades). The $1 backreference will always be empty, so this will always redirect to the home page (as you later found out).
The $1 backrefence will always be empty because you have no capturing groups in the RewriteRule pattern. It would have needed to be something like:
RewriteRule (.*) https://www.example.com/$1 [L,R=301]
The $1 then refers to whatever is captured by the (.*) subpattern.
Why did the earlier tests work... well, that's a puzzle. But be wary that 301 redirects are cached hard by the browser, so if you'd previously tested an alternative directive then you may have been seeing a cached response? Make sure the browser cache is clear before testing (test with the "object inspector" open and caching disabled). Also test with 302 (temporary) redirects so you don't end up caching an erroneous redirect.
what really is the best option to do all HTTP-to-HTTPS redirect?
Well, that's the thing, there is no single "best" option that will work on every server. It depends, on your set up.
Are you already canonicalising the www vs non-www? (This should often be done at the same time.)
Note that unless you have "UseCanonicalName On" set in the server config (which is not the default) then SERVER_NAME is the same as HTTP_HOST, ie. whatever hostname has been specified on the request. So, if you use SERVER_NAME (or HTTP_HOST) in the
substitution then you are going to redirect to the same (ie. example.com to example.com and www.example.com to www.example.com). So, in this repsect, hardcoding the canonical hostname in the
substitution (as you did in the first example) could be considered "better". For example:
RewriteRule ^ https://www.example.com%{REQUEST_URI} [R=301,L]
However, if "UseCanonicalName On" is set in the server config, then SERVER_NAME refers to the value set in the ServerName directive (in the server config), so this probably would be OK (ie. canoncialise the hostname), but that still depends on how your server is configured. (And note that this is not the default option, so it must be explicitly set in the server config.)
Some comparisons with the 3 rules you posted...
!=on
!=on [NC]
!on
The net result of all these are the same. The value of the HTTPS variable is always lowercase (barring any server peculiarities), so the NC flag is entirely superfluous (and just adds complexity). The value is either "on" or "off", so the "=" is optional. (The "=" prefix makes it into a string comparison, instead of a regex, so it's arguably "better" for performance - but that is insignificant.)
RewriteRule ^ https://www.example.com/$1 [L,R=301]
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [R=301,L]
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
As mentioned above, the first one won't work as intended. The second one is OK, but (probably) doesn't canonicalise the hostname. The third one has a superfluous capturing group in the RewriteRule pattern and will result in a temporary (302) redirect, instead of a 301.
If you implemented this redirect directly in the server config, the directives using REQUEST_URI would work without alteration. However, if you are using a backreference then the directive would need to be modified to avoid a double slash. (Although if you were doing this in the server config then you probably shouldn't be using mod_rewrite for this anyway.) (By "server config", I'm referring to a
server or
virtualhost context, not a
directory context.)
