Attributed metadata history

Preparing to Issue 200 Million Certificates in 24 Hours

On a normal day Let’s Encrypt issues nearly two million certificates. When we think about what essential infrastructure for the Internet needs to be prepared for though, we’re not thinking about normal days. We want to be prepared to respond as best we can to the most difficult situations that might arise. In some of the worst scenarios, we might want to re-issue all of our certificates in a 24 hour period in order to avoid widespread disruptions. That means being prepared to issue 200 million certificates in a day, something no publicly trusted CA has ever done. We recently completed most of the work and investments needed to issue 200 million certificates in a day and we thought we’d let people know what was involved. All of this was made possible by our sponsors and funders, including major hardware contributions from Cisco, Thales, and Fortinet. Scenarios Security and compliance events are a certainty in our industry. We obviously try to minimize them, but since we anticipate they will happen, we spend a lot of time preparing to respond in the best ways possible. In February of 2020 a bug affecting our compliance caused us to need to revoke and replace three million active certificates. That was approximately 2.6% of all active certificates. What if that bug had affected all of our certificates? That’s more than 150 million certificates covering more than 240 million domains. What if it had also been a more serious bug, requiring us to revoke and replace all certificates within 24 hours? That’s the kind of worst case scenario we need to be prepared for. To make matters worse, during an incident it takes some time to evaluate the problem and make decisions, so we would be starting revocation and reissuance somewhat after the beginning of the 24-hour clock. That means we would actually have less time once the momentous decision has been made to revoke and replace 150 million or more certificates. Infrastructure Improvements After reviewing our systems, we determined that there were four primary bottlenecks that would prevent us from replacing 200 million certificates in a day. Database performance, internal networking speed, cryptographic signing module (HSM) performance, and bandwidth. Database Performance Let’s Encrypt has a primary certificate authority database at the heart of the service we offer. This tracks the status of certificate issuance and the associated accounts. It is write heavy, with plenty of reads as well. At any given time a single database server is the writer, with some reads directed at identical machines with replicas. The single writer non-clustered strategy helps with consistency and reduces complexity, but it means that writes and some reads must operate within the performance constraints of a single machine. Our previous generation of database servers had dual Intel Xeon E5-2650 v4 CPUs, 24 physical cores in total. They had 1TB memory with 24 3.8TB SSDs connected via SATA in a RAID 10 configuration. These worked fine for daily issuance but would not be able to handle replacing all of our certificates in a single day. We have replaced them with a new generation of database server from Dell featuring dual AMD EPYC 7542 CPUs, 64 physical cores in total. These machines have 2TB of faster RAM. Much faster CPUs and double the memory is great, but the really interesting thing about these machines is that the EPYC CPUs provide 128 PCIe4 lanes each. This means we could pack in 24 6.4TB NVME drives for massive I/O performance. There is no viable hardware RAID for NVME, so we’ve switched to ZFS to provide the data protection we need. Internal Networking Let’s Encrypt infrastructure was originally built with 1G copper networking. Between bonding multiple connections for 2G performance, and use of the very limited number of 10G ports available on our switches, it served us pretty well until 2020. By 2020 the volume of data we were moving around internally was too much to handle efficiently with

This record contains metadata and attribution only. It does not host, download, unpack, render, or execute malware samples.