More than 83K certs from nearly 7K DigiCert customers must be swapped out now

Small stay of execution in 'exceptional circumstances' promised – amid legal action to pause digital bonfire

As the DigiCert drama continues, we now have a better idea of the size and scope of the problem – with the organization's infosec boss admitting the SSL/TLS certificate revocation sweep will affect tens of thousands of its customers, some of which have warned that the short notice may have real-world safety implications and disrupt critical services.

A brief refresher on what happened: On July 29, the certificate authority said at least some customers had just 24 hours to replace their previously issued security certificates due to a five-year-old programming flaw in its systems. It's fairly technical, and you can read all about it here in our earlier article.

It basically involves broken domain ownership validation, random numbers, and underscores, all leading to a selection of issued certs being deemed untrustworthy and thus in need of immediate revocation and replacement.

This affected "approximately 0.4 percent of the applicable domain validations" DigiCert had already processed for its customers, according to the CA. But it didn't put a number on this 0.4 percent.

A certain Mozilla Bugzilla post, however, puts the number in the tens of thousands. "We have identified 83,267 certs impacting 6,807 subscribers," noted DigiCert's CISO Jeremy Rowley. "We are planning to begin revoking within the 24-hour time window."

Some of these customers, however, say the quick turnaround time won't work. Others, according to Rowley, have filed legal challenges to stall the certificate revocation.

(DigiCert's PR representatives have told us that although Rowley indicated multiple customers had initiated legal action, there is so far only one such challenge and it was for a restraining order to hold off revocation for seven days.)

"Unfortunately, many other customers operating critical infrastructure, vital telecommunications networks, cloud services, and healthcare industries are not in a position to be revoked without critical service interruptions," Rowley added.

"While we have deployed automation with several willing customers, the reality is that many large organizations cannot reissue and deploy new certificates everywhere in time."

DigiCert will be publishing a full incident report, Rowley added, admitting that this whole snafu has illustrated the real-life difficulties that come with adhering to the 24-hour revocation requirements set by the CA/Browser Forum.

"We are aware of and are participating in the active industry discussion happening about the applicability of revocation timelines given the widespread impact and the relative severity of incidents," Rowley wrote.

"We also note that browsers have mentioned that delayed revocation might still be acceptable under 'exceptional circumstances.' However, given no clear definition of what would constitute an exceptional circumstance, we are seeking root store feedback as soon as possible, as we are standing ready to begin revocations within the timeline."

An email sent to affected customers this week and shared with The Register reiterated that some DigiCert users with "exceptional circumstances" would be given extra time — but not a whole lot of it.

"Based on discussions with these affected customers, conversations we have had with members of the web PKI community, we are now in a position to delay some revocations, given exceptional circumstances," the email explained.

Customers needing more time were encouraged to email the company by July 31, no later than 1930 UTC, with a detailed explanation of the circumstances necessitating a delay in the certificate renewal and revocation process. 

But even if a delay is approved, "All certificates affected by this incident, regardless of circumstances, will be revoked no later than Saturday, August 3, 2024, at 1930 UTC," the notice said. "We will not be able to delay revocation beyond that date and time."

One affected customer told The Reg they had 96 serial numbers of certificates to replace, and that required the IT team to update certs on almost 200 systems and applications.

"They told us via email on Monday, July 29 that we had until July 30 to swap out all the certificates before they were revoked," the reader told us. "It took 15 people 20 hours to touch everything. Good thing we noticed the email right away or it would have crushed us."

Our tipster wanted to remain anonymous to avoid putting their organization at risk. But they also wanted "to emphasize that for some organizations including ours, revoking certificates with such short notice can pose a risk to life and safety systems."

The reader continued:

Not every one of these certificate updates can be done Not every certificate update can be automated, and not every organization can manually replace their certificates within 24 hours (we only got 20 hours of notice to do almost 200 certificate updates). Often, certificate updates require collaboration with third parties, and not all organizations have support agreements that guarantee the necessary engagement within such a short timeframe. While I appreciate the importance of strict adherence to security protocols regarding certificates, a 24-hour notice before revocation is likely to cause more harm than good. Each incident of this scale should be evaluated on a case-by-case basis, rather than adhering to the one-size-fits-all 24-hour mandate set by the CA/Browser Forum.

This is unlikely to be the last we hear of the 24-hour rule — and the resulting thousands of organizations now scrambling to adhere to it and ensure secure internet communications. Meanwhile, hats off to the IT teams working overtime to manually swap out the flawed certificates with new ones.

Hopefully you get more than a $10 Uber Eats gift card for your long nights ahead. ®

Tell your story. Contact us in confidence, directly here or via here.

Editor's note: This story was updated to include extra context from DigiCert regarding the legal action it faces.