If your website or other online service uses other applications or integrations such as APIs, сURL, OpenSSL, etc. you may have experienced problems or outages. If you are still seeing any service disruptions or errors or your visitors use browsers older than 2015 and report issues, please contact us!
Let’s start with an explanation of how SSL certificates are used by various applications. Whenever an application (for example, a browser) contacts a web service over the SSL/TLS protocol, the web service provides a set of SSL certificates to the application. The application then verifies that they were issued for the service the application is accessing, that the expiration date of the certificates has not passed, and that the certificates were signed by a trusted Certificate Authority.
In order to verify the latter, the application attempts to link the provided certificates to one of the certificates contained in its trust storage. This trust storage is distributed either together with the operating system, runtime system, browser, or the application itself. If all the checks are passed, the application continues communication over the secure protocol. If not, the application either drops the connection or informs the user about potential security threats.
Sectigo’s AddTrust External CA Root was valid for 20 years until May 30, 2020 and was considered to be legacy. Using cross-certification, the Certificate Authority issued a pair of new Root certificates in 2010, which are valid until 2038, to replace the legacy Root. The new Root certificates were distributed via security updates to the majority of software applications utilizing the SSL/TLS protocol by mid-2015.
They continued to provide these certificates with the CA-bundles that included the AddTrust External CA Root and either USERTrust RSA Certification Authority or USERTrust ECC Certification Authority Intermediate (that expired on May 30, 2020) until April 30, 2020, to ensure that the certificates have the widest possible ubiquity (supported by most devices and systems, including the legacy ones).
As a result, we installed SSL certificates together with the CA-bundle that was due to expire before the end-entity certificate (the one issued for the domain name).
Thanks to the new cross-signed Root certificates, all modern browsers have both the expired and new Roots and automatically switched to using the new Root certificates. Users with these browsers did not experience any issues due to the expiration.
Apart from that, the expiration of the Root certificate did not put data transmitted over secured connections at risk. However, when it comes to applications other than browsers that use cURL, such as SSL/TLS libraries of various programming languages, and runtime systems, some were unprepared for the change.
These applications often use custom methods to verify SSL certificates. They may not rely on the operating system’s way of building the chain of trust and could not switch to using the new Root certificates. These applications either did not have the new Root certificates, had a broken certificate path validation logic, or were set up to explicitly trust the expired Root. For more detailed information on the affected clients, please refer to the research of Carnegie Mellon University.
The expiration of the Root certificate affected:
Legacy clients that did not receive security updates since before mid-2015
Clients configured to explicitly trust one of the expired Roots and ignore the operating system’s or vendor’s managed truststore
Client software based on OpenSSL library prior to version 1.1.1
Some OpenLDAP clients
Java applications that do not use the default truststore
Clients using cURL tool
Clients with configured alerting via Pingdom or OpsGenie
Applications that are connected to by any of the affected clients via SSL/TLS protocol
As a consequence of the expiration, various services were not able to initiate secure connections with other services or could not be accessed by other services.
Sectigo changed the CA-bundle provided to clients 30 days before the expiration of the root. But because of the bug in the systems of our upstream provider, they continued providing the AddTrust External CA Root until the date of its expiration, May 30.
This bug has already been taken care of and all customers will now receive a modern chain when we reissue the SSL to their sites.
We’ve checked all of our active certificates for a CA certificate in the chain that had expired, and asked our upstream provider to reissue the correct certificate bundles. This has been done for all, still active/valid certificates, that have been provided by us.
Our upstream SSL Certificate provider has implemented checks that will issue warnings to let us know a scenario like this will happen again before it actually happens.
We have been busy resolving this bit by bit since the problem started at the beginning of the week. The issue was confirmed completely resolved today.
Outages disrupt your life and your business. We understand and we take our responsibility to you very seriously.
Please allow me to take this opportunity to thank you for your business and provide my personal assurance that we are dedicated to meeting our commitment to you.
Sincerely,
Erlend Eide CEO