The DST Root CA X3, one of the root certificates used by LetsEncrypt, expired on September 30th. A majority of services and devices require no additional changes but some, particularly running OpenSSL v1.0.1, would require changes.

Applies to

• Linux installations operating as a client connecting to an endpoint protected by LetsEncrypt SSL
• OpenSSL v1.0.1 and older
• Has the new ISRG_Root_X1 certificate

To confirm that you have the ISRG_Root_X1 certificate run the following command:

find /usr/share/ca-certificates/ -name "ISRG_Root_X1*"
/usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt

Note: If you are on Ubuntu 12.04, use Ubuntu 12.04 client - LetsEncrypt DST_Root_CA_X3 expiry - OpenSSL fix as it doesn't have the new ISRG_Root_X1 certificate.

Note: If you are running the server with the LetsEncrypt certificate, you should contact your clients to apply these changes.

Background

The expiry of the DST_Root_CA_X3 was long-anticipated and the good folks of LetsEncrypt have been preparing us for this day for a long time. This change would not affect a majority of services but some legacy systems running OpenSSL v1.0.1 are affected. Such a system running a Ruby on Rails application caused an error when accessing an API exposed by this site.

Since I have attended 3 Ubuntu installations all of which were running OpenSSL v1.0.1. This post assumes you already have the ISRG_Root_X1 certificate in your system. The reason why OpenSSL v1.0.1 doesn't use it appears to be related to a known bug. People, always keep your software up-to-date.

The Error

OpenSSL::SSL::SSLError
SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed /home/deploy/.rbenv/versions/2.1.0/lib/ruby/2.1.0/net/http.rb:920:in `connect'
/home/deploy/.rbenv/versions/2.1.0/lib/ruby/2.1.0/net/http.rb:920:in `block in connect'
/home/deploy/.rbenv/versions/2.1.0/lib/ruby/2.1.0/timeout.rb:82:in `block in timeout'
/home/deploy/.rbenv/versions/2.1.0/lib/ruby/2.1.0/timeout.rb:70:in `catch'

Attempting to curl to a site also gives an error,

➜ curl https://www.jdeen.com
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

Instructions to fix

Step 1: Edit the certificates file at /etc/ca-certificates.conf with your favorite text editor. Mine is VIM.

sudo vim /etc/ca-certificates.conf

Step 2: Search for DST_Root_CA_X3.crt and comment on the line by adding ! at the beginning of the line. Once done it should look like this:

!mozilla/DST_Root_CA_X3.crt

Save and exit.

The reason why we comment on the expired certificate is because of the bug I mentioned to you. It only happens when the expired certificate is available to OpenSSL.

Step 3:

Now run the update-ca-certificates command. This will read the configuration file and generate one ca-certificates.cert file.

sudo update-ca-certificates

Confirmation

That is it! You can do a curl to confirm it is working:

curl https://www.jdeen.com

Extra 1: Some forum threads suggested the removal of the old certificate (we commented it out). I don't like this idea, but if you are to remove the file:

sudo rm /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt