Blocking IPs in firewall - a warning

letsencrypt uses several IP ranges

[dstiles]

Like some others hereabouts, I block large swathes of server farm IPs. Until recently this has been done via a database list read on site access - it still is on my Windows server and is a fall-back on my linux Apache server.

I recently decided to firewall a few persistent, high-incidence IP ranges on my linux server. Not entirely a good idea, hence this warning.

I found that suddenly, two or three times a day, letsencrypt updated ALL of the site configs on the server, with no actual changes to them. Looking in the letsencrypt log I discovered there were several traces per call indicating an error, with a comment that the cause may be due to firewalled IPs. Pulling out several IP ranges from the firewall fixed the problem. The ranges included Amazon, Microsoft and Digital Ocean, chosen from previous observation of their bot.

Let's Encrypt does not publish the IP ranges used: understandable in several ways. Now that they are visible in the logs again I could work out the CURRENT IP ranges but it's not a reliable method, as they say the IPs will change. As far as I can tell they use at least three cloud services and possibly, now or in the future, others.

Be warned: by all means firewall small service ranges but be wary of killing the bigger ones. Leave it to the site trappers such as databases and htacess.

[JorgeV]

Hello,

Indeed, Let's encrypt uses different IPs, this is stated in their FAQ:

[letsencrypt.org...]

What IP addresses does Let’s Encrypt use to validate my web server?
We don’t publish a list of IP addresses we use to validate, and these IP addresses may change at any time. Note that we now validate from multiple IP addresses.

and

Multi-Perspective Validation Improves Domain Validation Security
[letsencrypt.org...]

[SumGuy]

I manually update my lets encrypt certs. Do I need to worry about unintentionally blocking lets encrypt between certifications?

[dstiles]

Probably. As far as I know letsencrypt checks the site is available and that it can access the config file for it periodically; also that it can write its own logs, but this may only apply to auto-updated certs.

I assume you have a reason for manually updating, but I find it's safer and more reliable to let it carry on by itself - providing I haven't stupidly blocked it. :)

[SumGuy]

The version of Abyss I have doesn't handle auto-recertification (or if it does I havent figured it out). I run some windows command-line program (name escapes me right now) that interacts with lets encrypt and writes the required files to my site directory. I take the keys and cut/paste them into the Abyss user interface. Takes a few minutes. I only have to do this with 1 cert. I think the last time I did this the program failed - I was blocking lets encrypt. I checked my logs, found the IP and unblocked it, and it worked when I tried it again. I'll probably have to do that again next time.

I was just wondering why lets encrypt needs access to my site in between these manual certifications and what the consequences are if indeed they are blocked.

[dstiles]

LE checks daily to see if a cert needs to be updated - at least, I think that's why it does it. The LE site would be a good place to check, together with its forum.

On linux apache I automate using certbot. I forget the name of the windows one I use but that also works unattended.

[JorgeV]

Hello,

LE checks daily to see if a cert needs to be updated

Are you sure? Isn't it your own cron job, which is daily checking which certs need to be updated?

[dstiles]

You are correct, but I'm seeing a probe from LE. For example, a double hit to the acme directory every day (404 because no file).

Although on closer look it always checks the same web site (I have a dozen or so on the same IP, all with LE certs). That's odd, I admit. The cert is about a month into its life, so not expected to be updated for another few weeks. There is no well-known folder for it to view (hence the 404) but only one of the other sites has one so that's not unusual; I think perhaps it creates and deletes it as needed.

The probe is always at 9am BST. Normally, as I say, two probes from a viawest IP, but today it was preceded, at the same time to the second, by a double hit from amazon, again to the same site. Per day, it seems to be asking for the same challenge file but that varies per day.

[JorgeV]

This is interesting. I don't observe this on my sites. I wonder what can cause this.

[dstiles]

This seems, after a question on the letsencrypt forum, to result from migrating the site from another server and not cancelling that server's certificate for the site.

[JorgeV]

Okay ! Good to know.

[Brett_Tabke]

That is a great warning. I did the same thing with our PCI scanner by accident and it took 3 months to figure out. Meanwhile they were threatening to pull my gateway access to process cards.