Skip to content

fm: add certificate diagnosis engine for expiring silo TLS certificates #11238

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
smklein merged 15 commits into from
Oct 19, 2026
Merged

fm: add certificate diagnosis engine for expiring silo TLS certificates #11238

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
5007ba6
fm: add certificate diagnosis engine for expiring silo TLS certificates
smklein Sep 4, 2026
e8c3674
fm: name only the absolute expiration time in certificate fact comments
smklein Sep 4, 2026
590fba9
fm: only observe certificates Nexus can actually serve
smklein Sep 4, 2026
4d7489d
fm: fix wrapped warning string in the certificate loader
smklein Sep 4, 2026
9e449d0
fm: break best-certificate ties deterministically
smklein Sep 4, 2026
391d604
nexus: pick the best certificate by its converted not_after
smklein Sep 4, 2026
2e1ca73
fm: make certificate fact payload columns NOT NULL
smklein Sep 4, 2026
de88ede
fm: use silo_list_all_batched in the certificate loader
smklein Sep 4, 2026
4a64593
fm: make the parsed certificate case's fact non-optional
smklein Sep 4, 2026
cf29820
fm: address review feedback on the certificate diagnosis engine
smklein Sep 5, 2026
a723b3b
fm: make the shared best_certificate function concrete
smklein Sep 8, 2026
d5b30c8
merge
smklein Oct 15, 2026
bbb0900
fm: record the config diagnosis engines ran with in the input report
smklein Oct 15, 2026
2c519b5
fm: clarify ConfigReport docs
smklein Oct 15, 2026
8503c31
fm: clarify which config settings ConfigReport skips
smklein Oct 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions certificates/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ license = "MPL-2.0"
workspace = true

[dependencies]
chrono.workspace = true
display-error-chain.workspace = true
foreign-types.workspace = true
openssl.workspace = true
Expand Down
100 changes: 99 additions & 1 deletion certificates/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,15 @@

//! Utilities for validating X509 certificates, used by both nexus and wicketd.

use chrono::DateTime;
use chrono::Utc;
use display_error_chain::DisplayErrorChain;
use omicron_common::api::external::Error;
use openssl::asn1::Asn1Time;
use openssl::asn1::Asn1TimeRef;
use openssl::pkey::PKey;
use openssl::x509::X509;
use openssl::x509::X509Ref;
use std::borrow::Borrow;
use std::ffi::CString;

Expand Down Expand Up @@ -47,6 +51,12 @@ pub enum CertificateError {
#[error("Unsupported certificate purpose (not usable for server auth)")]
UnsupportedPurpose,

#[error("Certificate validity time could not be interpreted")]
BadValidityTime(#[source] openssl::error::ErrorStack),

#[error("Certificate validity time is out of the representable range")]
TimeOutOfRange,

#[error("Unexpected error")]
Unexpected(#[source] openssl::error::ErrorStack),
}
Expand All @@ -62,7 +72,9 @@ impl From<CertificateError> for Error {
| InvalidValidationHostname(_)
| ErrorValidatingHostname(_)
| NoDnsNameMatchingHostname { .. }
| UnsupportedPurpose => Error::invalid_value(
| UnsupportedPurpose
| BadValidityTime(_)
| TimeOutOfRange => Error::invalid_value(
"certificate",
DisplayErrorChain::new(&error).to_string(),
),
Expand All @@ -77,6 +89,54 @@ impl From<CertificateError> for Error {
}
}

/// The validity window of an X509 certificate.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub struct CertificateValidity {
/// The certificate is not valid before this time.
pub not_before: DateTime<Utc>,
/// The certificate is not valid after this time.
pub not_after: DateTime<Utc>,
}

/// Returns the validity window of an X509 certificate.
///
/// When called on the leaf certificate of a chain (the first certificate in
/// the chain, which is the one presented to TLS clients), the returned window
/// is the one clients will check.
pub fn validity(
cert: &X509Ref,
) -> Result<CertificateValidity, CertificateError> {
Ok(CertificateValidity {
not_before: asn1_time_to_chrono(cert.not_before())?,
not_after: asn1_time_to_chrono(cert.not_after())?,
})
}

/// Converts an ASN.1 time to a `chrono` timestamp by measuring its offset
/// from the Unix epoch.
///
/// `Asn1TimeRef` offers no direct conversion to a Unix timestamp. Instead we
/// use `Asn1TimeRef::diff`, a binding to OpenSSL's `ASN1_TIME_diff`, which
/// computes the difference between two ASN.1 times as whole days plus
/// remaining seconds.
fn asn1_time_to_chrono(
time: &Asn1TimeRef,
) -> Result<DateTime<Utc>, CertificateError> {
const SECS_PER_DAY: i64 = 24 * 60 * 60;
let epoch = Asn1Time::from_unix(0).map_err(CertificateError::Unexpected)?;
// `epoch.diff(time)` computes `time - epoch`. OpenSSL fails this call if
// `time` is not a well-formed ASN.1 time, or if the difference cannot be
// represented (years far outside the range its calendar arithmetic
// supports). Either way the certificate carries a validity time we
// cannot interpret.
let diff = epoch.diff(time).map_err(CertificateError::BadValidityTime)?;
let secs = i64::from(diff.days)
.checked_mul(SECS_PER_DAY)
.and_then(|s| s.checked_add(i64::from(diff.secs)))
.ok_or(CertificateError::TimeOutOfRange)?;
DateTime::from_timestamp(secs, 0).ok_or(CertificateError::TimeOutOfRange)
}

pub struct CertificateValidator {
validate_expiration: bool,
}
Expand Down Expand Up @@ -444,4 +504,42 @@ mod tests {
);
}
}

#[test]
fn test_validity_converts_asn1_times() {
// Give the leaf a specific validity window. The other certificates
// in the chain use rcgen's defaults, so the assertions below also
// check that we read the certificate we were given.
const NOT_BEFORE_SECS: u64 = 1_000_000_000;
const NOT_AFTER_SECS: u64 = 2_000_000_000;
let mut params = CertificateParams::new(vec![
"fake.test.oxide.computer".to_string(),
]);
params.not_before = (std::time::SystemTime::UNIX_EPOCH
+ std::time::Duration::from_secs(NOT_BEFORE_SECS))
.into();
params.not_after = (std::time::SystemTime::UNIX_EPOCH
+ std::time::Duration::from_secs(NOT_AFTER_SECS))
.into();
let chain = CertificateChain::with_params(params);
let certs = X509::stack_from_pem(chain.cert_chain_as_pem().as_bytes())
.expect("chain should parse");

let leaf_validity =
validity(&certs[0]).expect("leaf validity should convert");
assert_eq!(
leaf_validity,
CertificateValidity {
not_before: DateTime::from_timestamp(NOT_BEFORE_SECS as i64, 0)
.unwrap(),
not_after: DateTime::from_timestamp(NOT_AFTER_SECS as i64, 0)
.unwrap(),
}
);
// The intermediate certificate keeps rcgen's default window, which
// differs from the leaf's.
let intermediate_validity =
validity(&certs[1]).expect("intermediate validity should convert");
assert_ne!(leaf_validity, intermediate_validity);
}
}
20 changes: 20 additions & 0 deletions dev-tools/omdb/src/bin/omdb/nexus/fm_config.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,23 @@ struct ConfigOpts {
/// revert this setting to the default value.
#[clap(long, action = ArgAction::Set)]
analysis_enabled: Option<Setting<settings::AnalysisEnabled>>,

/// Sets the certificate expiry warning window, in days.
///
/// When a silo's external TLS certificate is within this many days of
/// expiring, the certificate diagnosis engine opens a case and requests an
/// alert.
///
/// The window applies to the certificate Nexus serves for the silo (the
/// one with the latest expiration time), so a case opens only when no
/// later-expiring replacement is installed.
///
/// This must be a non-zero integer no greater than 3650, or 'default'. If
/// it is set to 'default', any previous override will be removed, and the
/// system will revert this setting to the default value.
#[clap(long, action = ArgAction::Set)]
certificate_expiry_warning_days:
Option<Setting<settings::CertificateExpiryWarningDays>>,
Comment thread
smklein marked this conversation as resolved.
}

impl ConfigOpts {
Expand All @@ -151,6 +168,9 @@ impl ConfigOpts {
analysis_enabled: self
.analysis_enabled
.unwrap_or(current.analysis_enabled),
certificate_expiry_warning_days: self
.certificate_expiry_warning_days
.unwrap_or(current.certificate_expiry_warning_days),
}
}

Expand Down
98 changes: 56 additions & 42 deletions dev-tools/omdb/tests/successes.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -830,10 +830,11 @@ task: "fm_config_loader"
config last loaded at: <REDACTED_TIMESTAMP>
loaded by this activation: false
current config:
source: default
analysis enabled: true (default)
sitrep limit: 2500 (default)
history pruning threshold: 2000 (default)
source: default
analysis enabled: true (default)
sitrep limit: 2500 (default)
history pruning threshold: 2000 (default)
certificate expiry warning days: 30 (default)

task: "fm_rendezvous"
configured period: every <REDACTED_DURATION>m
Expand Down Expand Up @@ -887,6 +888,8 @@ task: "fm_sitrep_gc"
batches: 1
orphaned fm_ereport_in_case rows deleted: 0
batches: 1
orphaned fm_fact_certificate rows deleted: 0
batches: 1
orphaned fm_fact_physical_disk rows deleted: 0
batches: 1
orphaned fm_fact_saga rows deleted: 0
Expand All @@ -909,9 +912,10 @@ task: "fm_sitrep_history_pruner"
last completed activation: <REDACTED ITERATIONS>, triggered by <TRIGGERED_BY_REDACTED>
started at <REDACTED_TIMESTAMP> (<REDACTED DURATION>s ago) and ran for <REDACTED DURATION>ms
configuration:
analysis enabled: true (default)
sitrep limit: 2500 (default)
history pruning threshold: 2000 (default)
analysis enabled: true (default)
sitrep limit: 2500 (default)
history pruning threshold: 2000 (default)
certificate expiry warning days: 30 (default)
deletion batch size: 1000
status: within limit (nothing was deleted)
current count: 1
Expand Down Expand Up @@ -1568,10 +1572,11 @@ task: "fm_config_loader"
config last loaded at: <REDACTED_TIMESTAMP>
loaded by this activation: false
current config:
source: default
analysis enabled: true (default)
sitrep limit: 2500 (default)
history pruning threshold: 2000 (default)
source: default
analysis enabled: true (default)
sitrep limit: 2500 (default)
history pruning threshold: 2000 (default)
certificate expiry warning days: 30 (default)

task: "fm_rendezvous"
configured period: every <REDACTED_DURATION>m
Expand Down Expand Up @@ -1625,6 +1630,8 @@ task: "fm_sitrep_gc"
batches: 1
orphaned fm_ereport_in_case rows deleted: 0
batches: 1
orphaned fm_fact_certificate rows deleted: 0
batches: 1
orphaned fm_fact_physical_disk rows deleted: 0
batches: 1
orphaned fm_fact_saga rows deleted: 0
Expand All @@ -1647,9 +1654,10 @@ task: "fm_sitrep_history_pruner"
last completed activation: <REDACTED ITERATIONS>, triggered by <TRIGGERED_BY_REDACTED>
started at <REDACTED_TIMESTAMP> (<REDACTED DURATION>s ago) and ran for <REDACTED DURATION>ms
configuration:
analysis enabled: true (default)
sitrep limit: 2500 (default)
history pruning threshold: 2000 (default)
analysis enabled: true (default)
sitrep limit: 2500 (default)
history pruning threshold: 2000 (default)
certificate expiry warning days: 30 (default)
deletion batch size: 1000
status: within limit (nothing was deleted)
current count: 1
Expand Down Expand Up @@ -2579,10 +2587,11 @@ termination: Exited(0)
---------------------------------------------
stdout:
current fault management configuration:
source: default
analysis enabled: true (default)
sitrep limit: 2500 (default)
history pruning threshold: 2000 (default)
source: default
analysis enabled: true (default)
sitrep limit: 2500 (default)
history pruning threshold: 2000 (default)
certificate expiry warning days: 30 (default)

---------------------------------------------
stderr:
Expand All @@ -2593,9 +2602,10 @@ termination: Exited(0)
---------------------------------------------
stdout:
fault management config updated to version 1:
analysis enabled: true (default)
sitrep limit: 3000 (overriden)
history pruning threshold: 2000 (default)
analysis enabled: true (default)
sitrep limit: 3000 (overriden)
history pruning threshold: 2000 (default)
certificate expiry warning days: 30 (default)

---------------------------------------------
stderr:
Expand All @@ -2606,13 +2616,14 @@ termination: Exited(0)
---------------------------------------------
stdout:
current fault management configuration:
source: version 1
modified at: <REDACTED_TIMESTAMP>
comment:
source: version 1
modified at: <REDACTED_TIMESTAMP>
comment:
I am altering the config. Pray I do not alter it further.
analysis enabled: true (default)
sitrep limit: 3000 (overriden)
history pruning threshold: 2000 (default)
analysis enabled: true (default)
sitrep limit: 3000 (overriden)
history pruning threshold: 2000 (default)
certificate expiry warning days: 30 (default)

---------------------------------------------
stderr:
Expand All @@ -2623,9 +2634,10 @@ termination: Exited(0)
---------------------------------------------
stdout:
fault management config updated to version 2:
analysis enabled: false (overriden)
sitrep limit: 3000 (overriden)
history pruning threshold: 2000 (default)
analysis enabled: false (overriden)
sitrep limit: 3000 (overriden)
history pruning threshold: 2000 (default)
certificate expiry warning days: 30 (default)

---------------------------------------------
stderr:
Expand All @@ -2636,13 +2648,14 @@ termination: Exited(0)
---------------------------------------------
stdout:
current fault management configuration:
source: version 2
modified at: <REDACTED_TIMESTAMP>
comment:
source: version 2
modified at: <REDACTED_TIMESTAMP>
comment:
oops i altered it further
analysis enabled: false (overriden)
sitrep limit: 3000 (overriden)
history pruning threshold: 2000 (default)
analysis enabled: false (overriden)
sitrep limit: 3000 (overriden)
history pruning threshold: 2000 (default)
certificate expiry warning days: 30 (default)

---------------------------------------------
stderr:
Expand All @@ -2653,13 +2666,14 @@ termination: Exited(0)
---------------------------------------------
stdout:
fault management configuration v1:
source: version 1
modified at: <REDACTED_TIMESTAMP>
comment:
source: version 1
modified at: <REDACTED_TIMESTAMP>
comment:
I am altering the config. Pray I do not alter it further.
analysis enabled: true (default)
sitrep limit: 3000 (overriden)
history pruning threshold: 2000 (default)
analysis enabled: true (default)
sitrep limit: 3000 (overriden)
history pruning threshold: 2000 (default)
certificate expiry warning days: 30 (default)

---------------------------------------------
stderr:
Expand Down
Loading
Loading