Back on the chain gang
UPDATED The expiration of Let’s Encrypt's root certificate last week threw up a number of problems, though not perhaps in the areas predicted ahead of the event.
A legacy certificate used by the certificate authority – the IdentTrust DST Root CA X3 – expired on September 30.
Let’s Encrypt saw this issue coming more than two years ago, repeatedly advising its community and subscribers on how to move over to a new root cert.
BACKGROUND Device 'breakage' concerns persist days before Let’s Encrypt root cert expiry
The non-profit delayed the transition earlier this year in order to allow updates for Android to become available, and in the interests of making the changeover as smooth as possible.
Digital SSL certificates that underpin the HTTPS protocol are issued by trusted certificate authorities, with indexes updated through operating system updates.
If these indexes have not been updated, then affected systems will fail to recognize the new Let’s Encrypt root certificate – thereby breaking the chain of trust between a website and a user’s browser.
By way of example, the AddTrust External CA Root expired in May 2020, leaving multiple organizations with problems as a result.
Unchained melody
Last week's Let’s Encrypt root cert change also proved to be a bumpy road.
Issues arose not so much because of clients running obsolete versions of operating systems, but because the servers of several organizations failed to serve updated certificate chains to clients due to configuration problems.
Web security expert Scott Helme – who posted a detailed blog post warning of potential trouble with the Let’s Encrypt root cert shortly before the event – helped us survey the wreckage.
Ahead of the event, Helme had predicted that slightly older devices and software, particularly systems depending on OpenSSL 1.0.2, could be at risk. Let’s Encrypt previously expressed concerns about older Android devices.
Chain of fools
As it turned out, systems depending on OpenSSL 1.02 (which has been obsolete since December 2019) accounted for the majority of issue with the Let’s Encrypt root cert transition logged so far.
Big companies with an issue included: Palo Alto, Bluecoat, Cisco Umbrella, Catchpoint, Guardian Firewall, Monday.com, PFsense, Google Cloud Platform, Microsoft Azure Application Gateway, OVH, Auth0, Shopify, Xero, QuickBooks, Fortinet, Heroku, Rocket League, InstaPage, cPanel, Ledger, Netlify, Cloudflare Pages, Sophos, AWS, and DigitalOcean.
OpenBSD issued an alert on the Let’s Encrypt issue.
Read more of the latest browser security news
Most of the issues might be categorized as glitches, and the problem is largely in hand. However, anything that affects the fragile chain of trust that underpins e-commerce and much else ought to be a concern.
Issues for Fortinet users are more gnarly than the norm, even though the firm has since offered a workaround, as detailed in an advisory from the network security appliance firm:
The issue being seen by Fortinet customers is due to Fortinet devices validating the full chain of trust and then invalidating the chain when it sees that the CA IdenTrust DST Root CA X3 is expired, even though the cross-signed ISRG Root X1 root is valid for longer.
We have removed the offending expired certificate from the certificate store, however, this still does not solve the problem due to the Authority Information Access - CA Issuers entry.
The networking equipment firm concluded: “Fortinet is working on a longer-term solution to improve certificate validation and add additional intelligence to rebuild missing certificate chains in these cases going forward, and will include this in a future release.”
Unforced error
Many of the issues that Fortinet and customers of other tech firms experience as a result of the Let’s Encrypt root certificate expiry were preventable.
“A lot of the orgs that saw failures was because they depend on software (mainly OpenSSL) that was either old or missing an option being set to allow it to continue working,” Helme told The Daily Swig.
“It certainly seems to be the culprit for the Google and AWS ones, probably the Microsoft issue too.”
Helme has created a test site to help identify issues with clients.
The Daily Swig asked Let’s Encrypt for an update on the situation but we’re yet to hear back.
In earlier correspondence, a spokesperson for Let’s Encrypt asked us to reiterate that the organization had been trying to spread the word about the root expiration for more than two years.
“Our first blog post on the topic was posted in April 2019,” the spokesperson said.
In a pinned Tweet, published on September 30, the same day as the root cert expiry, Let’s Encrypt said: “Our cross-signed DST Root CA X3 expired today. If you are hitting an error, check out fixes in our community forum. We're seeing higher than normal renewals, so you may experience a slowdown in getting your certificates.”
In response to queries from The Daily Swig, Let's Encrypt said some disruption was inevitable and offered pointers towards useful resources.
The expiration last week did lead to some people having trouble accessing some sites. The internet is complex and many parts of it have been around for decades so it is inevitable that some older clients and devices will be affected when a change like this happens.
We knew that this would be the case, which is why we appreciate you helping to get the word out last week so people would understand and know how to investigate fixing it.
Scott [Helme] has kept a good list of the companies that were affected and we've been standing by on our community forum to help any of these companies or others who have come in looking for help on how to update or implement a workaround.
A blog post from Let's Encrypt summarizing resources can be found here.
This story has been updated with comment from Let's Encrypt
YOU MAY ALSO LIKE Protocol pollution vulnerabilities rife among high-traffic websites, study finds
