fm: add certificate diagnosis engine for expiring silo TLS certificates by smklein · Pull Request #11238 · oxidecomputer/omicron

smklein · 2026-09-04T16:59:02Z

Nexus serves each silo's external API with the silo's certificate whose leaf not_after is latest (ExternalEndpoint::best_certificate). When that certificate is about to expire or has expired, nothing told the operator.

This adds a fault management diagnosis engine (nexus/fm/src/diagnosis/certificate.rs) that opens a case per silo when the best certificate is inside a configurable warning window or already expired. The case carries one fact, BestCertificateExpiring or BestCertificateExpired, and requests a silo.certificate.expiring or silo.certificate.expired alert whenever that fact changes: entering the window, passing not_after, or a different (still expiring or expired) certificate becoming the best one. Unchanged input requests nothing. The case closes once a later-expiring certificate is installed or the silo is removed.

The warning window is a new FmConfig setting, certificate_expiry_warning_days (default 30, at most 3650). This value can be modified with omdb nexus fm-config set --certificate-expiry-warning-days <N>.

Supporting changes:

omicron-certificates: leaf_validity parses a PEM chain's leaf not_before/not_after into chrono timestamps.
nexus-types: DiagnosisEngineKind::Certificate, the CertificateFact payloads, the ObservedSiloCertificates analysis input, the two alert classes with V0 payloads, the FmConfig setting, and an input report section listing observed silos and certificates.
Database changes (schema/crdb/fm-certificate-de): diagnosis_engine and alert_class enum values, the fm_fact_certificate table (typed columns, one table per engine, as for disks and sagas), and the fm_config column with its CHECK constraint.
nexus-fm: Input now carries the FmConfig and the observed silo certificates (both required inputs). The fm_analysis task loads them from the same silo and certificate rows the external_endpoints task reads.

Not in scope, as follow-ups: not_before handling (to be added to best_certificate and the engine together), alerting on silos with no certificates at all, and exposing not_after in the external_endpoints task status.

Nexus serves each silo's external API with the silo's certificate whose leaf not_after is latest (ExternalEndpoint::best_certificate). When that certificate is about to expire or has expired, no later-expiring replacement is installed, and until now nothing told the operator. Add a fault management diagnosis engine that predicts the same choice and opens a case per silo when the best certificate is inside a configurable warning window or already expired. The case carries one fact (BestCertificateExpiring or BestCertificateExpired) and requests a silo.certificate.expiring or silo.certificate.expired alert whenever that fact changes. The case closes once a later-expiring certificate is installed or the silo is removed. Silos with no certificates open no case. Like best_certificate, the engine ignores not_before. The warning window is a new FmConfig setting, certificate_expiry_warning_days (default 30, at most 3650), settable via omdb nexus fm-config set --certificate-expiry-warning-days. Supporting changes: - omicron-certificates: leaf_validity parses a PEM chain's leaf not_before/not_after into chrono timestamps. - nexus-types: DiagnosisEngineKind::Certificate, CertificateFact payloads, the ObservedSiloCertificates analysis input, the two alert classes with V0 payloads, and the FmConfig setting. - Schema version 300 (fm-certificate-de): diagnosis_engine and alert_class enum values, the fm_fact_certificate table, and the fm_config column with its CHECK constraint. - nexus-fm: Input carries the FmConfig and observed silo certificates; the fm_analysis task loads them from the silo and certificate tables.

smklein · 2026-09-04T17:54:38Z

+    pub comment: String,
+
+    /// The silo this fact is about.
+    pub silo_id: Uuid,


Silos and certs don't have a strongly-typed UUID (yet), they probably should

A fact's comment is recorded when the fact is added and carried forward unchanged, so a relative phrase like "in 10days" or "1day ago" goes stale as sitreps advance. Record the absolute not_after only.

smklein · 2026-09-04T18:07:08Z

+    Ok(ParsedCertificateCase { silo_id, fact: kept, duplicate_facts })
+}
+
+pub(super) fn analyze(builder: &mut SitrepBuilder<'_>) -> anyhow::Result<()> {


A lot of this function is coping with "observing parent cases, reconciliation of facts". I'll annotate where the more important "new case opened, alert created" stuff happens.

smklein · 2026-09-04T18:07:44Z

+    for silo in silos.iter() {
+        let Some((desired, best)) = desired_fact(silo, reference_time, window)


This is where we flag certs which are "expired, or about to be"

smklein · 2026-09-04T18:10:55Z

+
+        let carried = parent.and_then(|(_, p)| p.fact.as_ref());
+        if carried.map(|(_, fact)| fact) == Some(&desired) {
+            continue;


we bail here if the fact was the same as the parent, so below this, the fact is new - meaning we need a new alert.

smklein · 2026-09-04T18:11:22Z

+            name: best.name.clone(),
+            not_after: best.not_after,
+        };
+        let alert_result = match &desired {


... and this is where we actually create alerts

hawkw · 2026-09-04T22:32:36Z

    }
+
+    #[test]
+    fn test_leaf_validity_reads_leaf_certificate() {


this feels like it might perhaps wanna be a proptest someday.

hawkw · 2026-09-04T22:49:36Z

+    // Expiry is judged against the input's deterministic "now"; see
+    // `Input::reference_time`.
+    let reference_time = input.reference_time();


thanks for adding this, i had been thinking we were gonna want something like this for a while now. (i was just gonna call it input.now(), but maybe your name is better!)

hawkw · 2026-09-04T22:57:25Z

(i should add that it's neat to see DEs for simple, but high-impact purely software/config issues starting to be written! thanks for doing this!)

The certificate diagnosis engine's loader accepted any certificate whose PEM parsed, but ExternalEndpoints::new also drops certificates whose key rustls cannot load. Such a certificate could be chosen as the silo's best and hide an expiring one that is served. Run the loader's rows through TlsCertificate::try_from so both sides agree.

The "silo not found" warning was wrapped without a trailing backslash, leaving a run of interior whitespace in the text omdb prints.

ObservedSiloCertificates::best_certificate claimed callers depended only on not_after, but the engine stores the chosen certificate's id in its fact and re-alerts when it changes. Break ties toward the greatest id, document why, and test that equal-expiry certificates carry the fact unchanged across sitreps.

TlsCertificate now carries the leaf validity as a DateTime, so the served-certificate rule can be the same max_by_key expression the certificate diagnosis engine uses, instead of a hand-rolled loop over Asn1Time partial_cmp. Drop the parsed X509 field, which had no other readers left.

Both certificate fact kinds carry the same payload, so the nullable columns plus two identical per-kind CHECK constraints were NOT NULL spelled twice. Declare them NOT NULL, drop the Option fields and the unreachable missing-column error path in the Diesel model.

The loader hand-rolled the same pagination loop the datastore helper already provides, with an ad-hoc batch size. Use the helper for silos and the standard SQL_BATCH_SIZE for certificates.

parse_case returns NoFacts when a case has no facts, so a successfully parsed case always has one. Store it directly and derive the silo id from it instead of threading an Option through the reconcile loop.

- Share certificate selection between Nexus and the engine: both ExternalEndpoint::best_certificate and ObservedSiloCertificates::best_certificate now call nexus_types::observed_certificate::best_certificate, and TlsCertificate carries the certificate id so the tie-break matches. The comments asking to keep the two in sync are gone, and a Nexus-side tie-break test is added. - Record the analysis reference time in InputReport. - omicron-certificates: map ASN1_TIME_diff failures to a new CertificateError::BadValidityTime (an invalid certificate, not an internal error), use checked arithmetic for the epoch offset, and clarify the docs and test comment. - Reword the engine's comments on parsing parent cases: a corrupt case is quarantined by closing it, at the cost of one duplicate alert if its silo still needs attention. - Restructure the omdb --certificate-expiry-warning-days help text.

Replace the generic helper with one that takes a slice of CertificateCandidate (id and not_after) and returns the winning id. Both Nexus and the certificate diagnosis engine build candidates from their own certificate types and look the winner up by id, so the whole selection rule lives in one non-generic function.

mergeconflict · 2026-09-15T19:04:33Z

+    /// the diagnosis engines for this analysis pass. Keyed by silo ID.
+    #[serde(default)]
+    pub observed_silo_certificates: BTreeMap<Uuid, SiloCertificatesReport>,
    // Reports are serialized to the database, so any new field here should


I think you need one more thing here, the current value of certificate_expiry_warning_days, which was input to the DE. Or some proxy for that value (e.g. maybe the config version that contained the value, although then it's no longer self contained)?

Addressed in bbb0900 - now we're emitting the configuration options to FM as part of the InputReport

mergeconflict · 2026-09-15T19:11:50Z

+            None => case_mut.close(format!(
+                "silo {} ({silo_id}) has no certificates installed; that \
+                 is outside this engine's scope",
+                silo.silo_name,


What about if the operator responds to a "certificate expired" alert by just deleting the certificate, leaving no remaining certs installed for some given silo? The DE says "yup, case closed," but now Nexus can't serve TLS for that silo at all.

smklein · 2026-10-15T23:08:30Z

This should be ready for a re-review

hawkw · 2026-10-15T23:55:08Z

I should probably add silos to the list of impact-able resources over in #11029

Fixes the `nexus-fm` test build on main. #11238 added a test that constructs `steno::SagaId(Uuid::new_v4())`, but #11517 (steno 0.5) landed first and made `SagaId` a newtype-uuid type alias, so the tuple-struct constructor no longer exists. Each PR passed CI on its own; the break only shows up once both are merged. This switches the test to `steno::SagaId::new_v4()`.

smklein marked this pull request as draft September 4, 2026 17:19

smklein commented Sep 4, 2026

View reviewed changes

fm: name only the absolute expiration time in certificate fact comments

e8c3674

A fact's comment is recorded when the fact is added and carried forward unchanged, so a relative phrase like "in 10days" or "1day ago" goes stale as sitreps advance. Record the absolute not_after only.

smklein commented Sep 4, 2026

View reviewed changes

smklein marked this pull request as ready for review September 4, 2026 18:43

smklein requested review from hawkw and mergeconflict September 4, 2026 19:01

smklein assigned augustuswm and unassigned augustuswm Sep 4, 2026

smklein requested a review from augustuswm September 4, 2026 19:01

smklein mentioned this pull request Sep 4, 2026

Alerts should have more granular scoping than "fleet" #11244

Open

hawkw reviewed Sep 4, 2026

View reviewed changes

smklein added 5 commits September 4, 2026 16:23

fm: fix wrapped warning string in the certificate loader

4d7489d

The "silo not found" warning was wrapped without a trailing backslash, leaving a run of interior whitespace in the text omdb prints.

This was referenced Sep 4, 2026

fm: extract shared keyed-case adoption scaffolding from the saga and certificate engines #11246

Open

fm: extract shared keyed-case adoption scaffolding from the saga and certificate engines #11247

Closed

smklein added 4 commits September 4, 2026 16:55

fm: use silo_list_all_batched in the certificate loader

de88ede

The loader hand-rolled the same pagination loop the datastore helper already provides, with an ad-hoc batch size. Use the helper for silos and the standard SQL_BATCH_SIZE for certificates.

fm: make the parsed certificate case's fact non-optional

4a64593

parse_case returns NoFacts when a case has no facts, so a successfully parsed case always has one. Store it directly and derive the silo id from it instead of threading an Option through the reconcile loop.

mergeconflict reviewed Sep 15, 2026

View reviewed changes

hawkw added the fault-management Everything related to the fault-management initiative (RFD480 and others) label Sep 18, 2026

smklein mentioned this pull request Oct 15, 2026

Prevent deleting a silo's last certificate #11510

Open

smklein added 2 commits October 15, 2026 15:43

merge

d5b30c8

fm: record the config diagnosis engines ran with in the input report

bbb0900

smklein added 2 commits October 15, 2026 16:00

fm: clarify ConfigReport docs

2c519b5

fm: clarify which config settings ConfigReport skips

8503c31

mergeconflict self-requested a review October 16, 2026 22:36

mergeconflict approved these changes Oct 16, 2026

View reviewed changes

smklein merged commit 79112b0 into main Oct 19, 2026
21 checks passed

smklein deleted the fm-certificate-de branch October 19, 2026 17:25

smklein mentioned this pull request Oct 19, 2026

fm: fix certificate DE test build with steno 0.5 #11526

Merged

		for silo in silos.iter() {
		let Some((desired, best)) = desired_fact(silo, reference_time, window)

Conversation

smklein commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hawkw commented Sep 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smklein commented Oct 15, 2026

Uh oh!

hawkw commented Oct 15, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

smklein commented Sep 4, 2026 •

edited

Loading