Let's Encrypt validation getting error 500

[TorontoBoy]

Ok, I've looked at this as much as the Wicked Witch of the West stared into her magic mirror, and I cannot find the problem. Let's Encrypt is GETting me daily, which is fine, but I return an error 500, which I should not.

66.133.109.3xx [29/Apr/2018:22:17:47 GET /.well-known/acme-challenge/_eaHuq8wjizuOgFLlkrN2gdFrCZsIRGyFZoaZbCt-N8 HTTP/1.1500 - - Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)

UA is Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)
On the other hand, my host provider sends me a very similar GET and I return a 200.

66.198.xxx.xxx [29/Apr/2018:22:17:36 GET /.well-known/acme-challenge/1K1OVFXXN803XYUDX10QDF907RYAII0C HTTP/1.1 200 64 - Cpanel-HTTP-Client/1.0

UA is Cpanel-HTTP-Client/1.0

I don't ban either IP range, nor am I consciously aware of banning the Let's Encrypt UA, or even parts of it. Can someone spot anything odd?

Thanks

[keyplyr]

Try adding this to your .htaccess:

Options +FollowSymlinks
RewriteRule ^.well-known/(.*)$ - [L]

[lucy24]

I don't ban either IP range, nor am I consciously aware of banning the Let's Encrypt UA, or even parts of it.

Irrelevant, fortunately. Or perhaps unfortunately, depending on what's ultimately going on. A ban would show up as a 403 (possibly some other 400-class error in some highly specialized circumstances); a 500 is a mistake. The question is whether it's your mistake or your host's. Does /.well-known/ live in the same place as the rest of your site, or is it aliased to some other location?

What do error logs say? A 500-class error should show something.

RewriteRule ^.well-known/(.*)$ - [L]

Or, if you prefer,

RewriteRule ^\.well-known/ - [L]

where the first change is a non-lethal error and the second is a matter for a different thread.

[keyplyr]

You'll see a 500 error because of the /.well-known/

Adding the code I gave will allow Let's Encrypt

[TorontoBoy]

My errors file, "Last 300 Error Log messages in reverse order", from cPanel has neither /.well-known/ nor the Let's Encrypt IP address. well-known errors in this Errors file, which I do regularly monitor, would have helped in pointing out the error.

I do have a /.well-known/ directory under public_html, which I really did not notice. Each of my sites are in their own subdirectory. I have repeatedly asked the tech support of my host provider about these Let's Encrypt errors, and they repeatedly say it is something within my htaccess. I don't have any rewriterule for .well-known in my htaccess.

Options +FollowSymlinks
RewriteRule ^.well-known/(.*)$ - [L]

What does this code do? I'm not sure what are symbolic links?
The rewriterule takes /.well-known/something, replaces it with "-" (pass it through unchanged?), stops processing rules and evalulate? Is that right?

[keyplyr]

What does this code do?

I do this with all the sites I work on as a proactive measure so I don't have to come back and figure out any issues. It allows Let's Encrypt access so it can install the cert's private key.

Adding the option directive to follow system links is usually unnecessary as well, but in your case it may serve well.

[lucy24]

What does this code do? I'm not sure what are symbolic links?
The rewriterule takes /.well-known/something, replaces it with "-" (pass it through unchanged?), stops processing rules and evalulate? Is that right?

Yes, more or less. The "-" here isn't really a replacement; it's a placeholder meaning “don’t change the original request” (because mod_rewrite syntax requires that you put something in the “target” position).

The rule needs to go before all other RewriteRules, so that nothing else has a chance to kick in. If it works, it means that some other RewriteRule was causing the problem (although I still don’t see why it would throw a 500). If there is no change, we will need to look elsewhere.

I have Let'sEncrypt certificates on two of my sites and didn’t need to make any htaccess changes, so clearly it depends on the individual server setup.

Following system links

Typo for “symbolic links”, right? This directive is required in order for mod_rewrite to work as intended in htaccess. But it is inherited--i.e. it is not, in itself, a mod_rewrite directive, confusing!--so any sane host will already have the directive in the appropriate part of their config file if they allow htaccess at all.

My fingers do not like typing the word “appropriate”.

[TorontoBoy]

Thanks for the explanation. I really like to understand all entries into the htaccess, as there may be side effects.

I put in the commands, but after my other RewriteRules and before my RewriteCond, SetEnvIf and IP bans. It did not work, so I have moved them up. No harm done. I'll need to monitor tomorrow.

[lucy24]

after my other RewriteRules and before my RewriteCond, SetEnvIf and IP bans

It has to be before all other RewriteRules, or it will have no effect. That was the whole point of the rule. An [L] flag affects only mod_rewrite; it has no effect on other mods.

Not sure what you mean by RewriteCond bans, since a RewriteCond belongs to an individual RewriteRule.

Each module is an island, so it doesn't matter what order you arrange the directives for different modules (mod_setenvif, mod_auththingy and so on). For your own sanity you will of course keep each mod's directives grouped together. On most servers, mod_setenvif executes early--reverse alphabetical order is a good rough guide--so environmental variables can be used by other mods.

[keyplyr]

Yes, the code I gave needs to go before all other RewriteRules.

I don't know why this has become so complicated.

[TorontoBoy]

I guess it worked. I received 2 requests and returned to them 200s, and then all activity from Let's Encrypt abruptly stopped since 2018 May 05. No further requests, no 200s, no 500s, nada. It is like I sent a message back to the mothership and they are leaving me alone? I was expecting a bit more communication.

66.133.109.36 [05/May/2018:08:16:47 GET /.well-known/acme-challenge/j7SDCPKfpvBP3bGGYf8ylVjiyCrZ4ztfzHniQPeBu1c HTTP/1.1 200 87 - Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)
66.133.109.36 [05/May/2018:08:16:47 GET /.well-known/acme-challenge/yKXs4pA_WUQycoCgzReohppBTtn-OKfubGtdU__m-CI HTTP/1.1 200 87 - Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)

On the assumption that Let's Encrypt is satisfied, thanks goes out to Keyplr and Lucy! Cheers.

[keyplyr]

Good work :)

[phranque]

you should endeavor to understand why the change worked for you and make sure it didn't introduce a new problem.

i'm not saying these particular suggestions are wrong - rather that you should know what every line in your .htaccess does and why it works.
it doesn't matter who suggested it - your specific situation may be different.

in this case you apparently bypassed an access block on that request.
have you now opened access to other UAs/IPs without that intention?

[TorontoBoy]

@phranque Thanks for your concern. It is one that I share. I have been monitoring my log to ensure that /.well-known/ has only been used by Let's Encrypt validation and IP and similar validation from my host provider's IP. I'll continue to monitor. I see the possible hole in my htaccess.

I am confident as to why the change worked, but do not know why the Let's Encrypt validation to suddenly and abruptly stop their requests. I'll need to do more research.

[keyplyr]

I see the possible hole in my htaccess

There's no possible hole caused by what I suggested. It does not *allow* anything that wasn't already allowed. It simply makes the directory easier to find by Let's Encrypt (as I said.)

This directory needs access from a number of agents: ISPs, newer browsers may start asking for this file for DNT, the various EU GDPR agents, Security Companies, some company firewalls evaluating security, etc.

Files commonly residing in the /.well-known/ [serverfault.com] :
• dnt-policy.txt [webmasterworld.com]
• security.txt [webmasterworld.com]
• pgp-key.txt [gist.github.com]
• assetlinks.json
• apple-app-site-association [webmasterworld.com]
• acme-challenge/ (used by Let's Encrypt)
and an increasing number of other files...

- - -

[TorontoBoy]

@keyplyr so the RewriteRule ^.well-known/(.*)$ - [L] is not a security risk? My host provider has only allowed Let's Encrypt in and should screen out others that could use the same GET request?

[keyplyr]

@keyplyr so the RewriteRule ^.well-known/(.*)$ - [L] is not a security risk?

No, as I said, it does not alter anything you already have in your htaccess.

My host provider has only allowed Let's Encrypt in and should screen out others that could use the same GET request?

I think you may be misunderstanding this code. It does not "allow" anything. It just helps Let's Encrypt find the directory. The directory is hidden and is not linked from your pages. The directory is already allowed, unless you are blocking IPs & UAs from access to your site as a whole.

Other agents need access to this directory also (read above.)

[phranque]

There's no possible hole caused by what I suggested. It does not *allow* anything that wasn't already allowed. It simply makes the directory easier to find by Let's Encrypt (as I said.)

more precisely it makes that directory accessible to requests by any and all user agents including let's encrypt.

it does not alter anything you already have in your htaccess.

it bypasses any subsequent mod_rewrite directives in that .htaccess and has no effect on directives from other modules.

Other agents need access to this directory also (read above.)

now you know...

[lucy24]

Files commonly residing in /.well-known/

The directory is hidden

I'm having trouble reconciling these two statements. If the directory is hidden then how would people ever be able to add files to it?

[keyplyr]

@lucy24 - Hosts/server accounts are set-up differently. /.well-known/ [serverfault.com] is a relatively new addition to a site's structure to augment TLS and Privacy.

As discussed in another thread, to *see* where I am managing some of these files that are supposed to be in the /.well-known/ directory, I actually put them somewhere else and 301. This is because of the way my host has structured accounts.

[TorontoBoy]

My "/.well-known" and nested "acme-challenge" dirs has file attributes 755, and within the acme-challenge the files have 644s, so mostly only reads. I don't know what goes into these folders, but it does not look executable. It seems pretty safe.