Skip to content

Expose SPIFFE client-auth registration through the operator CRDs - #6500

Open
jhrozek wants to merge 5 commits into
spiffe-integration-split3-5from
spiffe-integration-split3-6
Open

Expose SPIFFE client-auth registration through the operator CRDs#6500
jhrozek wants to merge 5 commits into
spiffe-integration-split3-5from
spiffe-integration-split3-6

Conversation

@jhrozek

@jhrozek jhrozek commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Summary

The SPIFFE client-authentication runtime model (trust domains + static workload-to-client associations) already exists in pkg/authserver, but was only reachable by hand-authoring an auth-server RunConfig file directly — there was no way to declare it through VirtualMCPServer/MCPExternalAuthConfig.

An earlier, never-reviewed attempt at this CRD exposure existed on an abandoned branch, but assumed a stale runtime shape (SPIFFE clients nested under inboundGrants.tokenExchange) that no longer matches the reviewed model (inboundGrants.spiffeClientAuth is now a sibling of tokenExchange/jwtBearer, with a larger field surface: a bundleSource discriminated union, resources, and a fixed grantTypes). This PR redoes the CRD exposure from scratch against the current model, reusing the old attempt's validation contract (its CEL test matrix) as an acceptance checklist, not its code.

  • Add spiffeTrustDomains and inboundGrants.spiffeClientAuth to the shared EmbeddedAuthServerConfig, with CEL admission validation for everything derivable from the object's own spec (paired configuration, no duplicate names/trust-domains/client-IDs/principal-patterns, full cross-referencing between domains and clients, method-subset enforcement, client-ID hygiene).
  • Wire the new fields through to authserver.RunConfig via the same converter pattern used for delegateClients/trustedIssuers, and extend reconcile-time revalidation to cover SPIFFE trust domains.
  • Deliberately do NOT expose grantTypes as a CRD field — the runtime only accepts exactly one value (urn:ietf:params:oauth:grant-type:token-exchange), so the converter sets it rather than exposing a field with one legal value.
  • Cross-field checks needing reconcile-time-derived values (allowedAudiences, scopesSupported — neither is CRD-exposed) are deliberately left to the existing reconcile-time revalidation path rather than a premature admission-time Go check — an earlier draft of this PR added such a check passing nil for both, which the runtime validator treats as "validate against nothing" rather than "skip," silently rejecting any resources entry or non-default scopes value. Caught and removed during adversarial review; see commit history for detail.

Refs #6199, #6200, #6205

Known, intentional consequence: a syntactically valid SPIFFE config is admitted by CEL but still fails reconciliation with a terminal error until RunConfig's "not yet enforced" placeholder gate (pkg/authserver/config.go) is lifted by a future PR — no live SVID verification exists yet, so this is expected, not a regression.

Type of change

  • New feature

Test plan

  • Unit tests (task test)
  • Linting (task lint-fix)

31-case CEL admission test matrix (spiffe_cel_test.go) ported from the old attempt's test list and extended for the bundleSource discriminated union and resources; unit tests for the two new converter functions; a reconcile-level test proving a SPIFFE client with resources/custom scopes is accepted at the admission-equivalent Go layer but still correctly validated (accepted or rejected as appropriate) once real derived values reach BuildAuthServerRunConfig.

API Compatibility

  • This PR does not break the v1beta1 API — both new fields are optional and additive.

Changes

Large diff (~3,384 lines), dominated by generated CRD YAML/docs (deploy/charts/operator-crds/**, docs/operator/crd-api.md — roughly 2,200 of the changed lines) and the new CEL test file. Hand-written surface: mcpexternalauthconfig_types.go (new CRD types + 11 CEL rules), controllerutil/authserver.go (converter wiring), and their tests.

Does this introduce a user-facing change?

Yes — operators can now declare spiffeTrustDomains/inboundGrants.spiffeClientAuth on VirtualMCPServer/MCPExternalAuthConfig. Reconciliation of a non-empty config will fail with a clear terminal error until live SVID verification lands (see "Known, intentional consequence" above).

Implementation plan

Approved implementation plan

Expose SPIFFE client-auth registration through the operator CRDs

Context

The SPIFFE client-auth epic (#6199, refs #6200/#6205) already landed a full
runtime model in pkg/authserver (SPIFFETrustDomainRunConfig,
SPIFFEClientAuthRunConfig, ValidateSPIFFETrust, the static-client
registry/storage overlay, etc.), but that model is only reachable by hand-
authoring an auth-server RunConfig file — there is no way to declare it
through VirtualMCPServer/MCPExternalAuthConfig CRDs.

An old, never-reviewed branch (spiffe-integration-split3-5/6/7, abandoned
mid-epic when the stack was reworked through review) had already built this
CRD-exposure feature once, but its commit (0c921b295, "Expose SPIFFE
registration in CRDs") assumed a runtime shape that no longer matches: it
nested SPIFFE clients under InboundGrants.TokenExchange.SPIFFEClients and
never modeled BundleSource/Resources/GrantTypes. The current, reviewed
runtime model puts SPIFFEClientAuth as a top-level sibling of
TokenExchange on InboundGrantsRunConfig (deliberately independent of the
token-exchange capability toggle — see the doc comment on
InboundGrantsRunConfig in pkg/authserver/spiffe_trust.go), and has a
materially larger SPIFFEClientAuthRunConfig/SPIFFETrustDomainRunConfig
surface than the old commit ever covered. Rather than merge that stale
commit, we're redoing the CRD-exposure work from scratch against the current
model, reusing the old commit's validation contract (its CEL test matrix)
as an acceptance checklist, not its code.

Known, intentional consequence: RunConfig.Validate() currently
hard-rejects any non-empty SPIFFETrustDomains via
validateSPIFFENotYetEnforced — a deliberate placeholder until real SVID
verification lands (see pkg/authserver/config.go:328-345). Wiring CRD
fields into RunConfig.SPIFFETrustDomains means a syntactically valid CRD
spec will be admitted by CEL/webhook validation but will still fail
reconciliation with a terminal InvalidEmbeddedAuthServerConfigError until
that gate is lifted by a future PR. This is expected, not a bug — call it
out explicitly in the PR description.

Scope

In scope: two new CRD types (SPIFFETrustDomainConfig,
SPIFFEClientConfig) plus a bundle-source discriminated union, added to the
shared EmbeddedAuthServerConfig (used by both MCPExternalAuthConfig and
VirtualMCPServer — no VMCP-specific code needed, same as the old commit
found). CEL admission validation for everything expressible from the
object's own spec. Go-level bridging into the existing
authserver.ValidateSPIFFETrust/NewSPIFFETrustConfig for everything that
needs derived values (AllowedAudiences, ScopesSupported — neither is
CRD-exposed; both are derived at reconcile time, confirmed via
cmd/thv-operator/pkg/controllerutil/authserver.go:883-895) or the full
SPIFFE-ID/URL grammar. Reconcile-time revalidation via the existing
validateDelegateClientsAndTrustedIssuers pattern.

Out of scope: bundle fetching/loading, live SVID verification, lifting
the validateSPIFFENotYetEnforced gate (all still placeholder by design),
and the legacy-field-removal refactor that was tangled into the old commit
(0c921b295) — that already landed separately via 0501c3fea /
split3-5.

CRD types — cmd/thv-operator/api/v1beta1/mcpexternalauthconfig_types.go

Mirror the runtime shape exactly (field-for-field with
SPIFFETrustDomainRunConfig/SPIFFEClientAuthRunConfig in
pkg/authserver/spiffe_trust.go), using the same validation-marker idioms as
neighboring types (DelegateClientConfig at line 361, TrustedIssuerConfig
at line 404) — +kubebuilder:validation:MinLength/MaxLength,
MinItems/MaxItems, +listType=atomic/set, +optional on every pointer
field, //nolint:lll on types carrying long CEL rules.

  1. SPIFFEAuthenticationMethod enum (spiffe_x509, spiffe_jwt),
    SPIFFEBundleSourceType enum (bundle_endpoint, workload_api),
    SPIFFEBundleEndpointProfile enum (https_web, https_spiffe) —
    straight string-const mirrors of the runtime enums.

  2. SPIFFEBundleSourceConfig (discriminated union, required field on
    SPIFFETrustDomainConfig — the runtime field has no omitempty):

    • Type SPIFFEBundleSourceType
    • Endpoint *SPIFFEBundleEndpointSourceConfig (URL string,
      Profile SPIFFEBundleEndpointProfile)
    • WorkloadAPI *SPIFFEWorkloadAPIBundleSourceConfig (empty struct)
    • Two XValidation rules reusing the exact pattern already used for
      MCPExternalAuthConfig's own type-discriminated union (line 67-75):
      self.type == 'bundle_endpoint' ? has(self.endpoint) : !has(self.endpoint)
      and the workload_api/workloadAPI mirror.
  3. SPIFFETrustDomainConfig: Name, TrustDomain (pattern-validated,
    copy the old commit's regex — it's a reasonable approximation and the
    type doc comment should say runtime parsing via spiffeid.TrustDomainFromString
    remains authoritative, matching the existing "CEL is best-effort" convention
    already used elsewhere in this file), Methods (MinItems=1,MaxItems=2,
    enum), BundleSource SPIFFEBundleSourceConfig (required, no +optional).

  4. SPIFFEClientConfig (lives in a new SPIFFEClientAuth []SPIFFEClientConfig field on InboundGrantsConfig — a sibling
    of TokenExchange/JWTBearer, not nested under TokenExchange, matching
    InboundGrantsRunConfig's shape):

    • TrustDomainRef, PrincipalPattern (pattern-validated + the old
      commit's path-traversal CEL rule
      self.principalPattern.split('/').all(segment, segment != '.' && segment != '..')),
      ClientID, Methods (MinItems=1,MaxItems=2), Audiences
      (MinItems=1,MaxItems=50), Scopes (MinItems=1,MaxItems=50),
      Resources (+optional, MaxItems=50 — new vs. the old commit; RFC
      8707 resources, Go-validated against AllowedAudiences since that's
      derived, not CRD-known).
    • Do not expose GrantTypes as a CRD field. The runtime only accepts
      exactly ["urn:ietf:params:oauth:grant-type:token-exchange"]
      (validateSPIFFEGrants, spiffe_trust.go:758-766) — a field with one
      legal value is a field the converter should just set, not a knob users
      configure. Set it in the converter (GrantTypes: []string{authserver.SPIFFEGrantTypeTokenExchange}).
  5. SPIFFETrustDomains []SPIFFETrustDomainConfig stays on
    EmbeddedAuthServerConfig directly (top-level, matching
    RunConfig.SPIFFETrustDomains) — MinItems=1,MaxItems=50, +listType=atomic, +optional, same placement the old commit used.

  6. CEL rules on EmbeddedAuthServerConfig (add alongside its existing
    XValidation block): reuse the old commit's semantic set, adjusted for
    the new field path (self.inboundGrants.spiffeClientAuth, not
    self.inboundGrants.tokenExchange.spiffeClients):

    • paired configuration (spiffeTrustDomains and
      inboundGrants.spiffeClientAuth empty/non-empty together)
    • no duplicate trust-domain name
    • no duplicate trust-domain trustDomain value
    • every declared trust domain referenced by ≥1 client
    • every client's trustDomainRef resolves to a declared domain
    • client methods ⊆ referenced domain's methods
    • principalPattern's trust-domain segment matches the referenced
      domain's trustDomain
    • no duplicate client clientId
    • no duplicate client principalPattern
    • clientId must not start with synthetic: (reserved) or look like an
      absolute URL (reserved for CIMD) — reuse
      storage.ValidateRegisterableClientID's intent as the CEL approximation,
      Go-side stays authoritative

Go bridging — same file, new validateSPIFFETrustConfig helper

Alongside the existing validateEmbeddedAuthServer(): convert
cfg.SPIFFETrustDomains[]authserver.SPIFFETrustDomainRunConfig and
cfg.InboundGrants.SPIFFEClientAuth[]authserver.SPIFFEClientAuthRunConfig
(setting GrantTypes to the fixed value here), call
authserver.ValidateSPIFFETrust(trustDomains, inboundGrants, nil, nil) for a
config-shape-only check at admission time (no AllowedAudiences/
ScopesSupported available yet — same limitation already accepted for
DelegateClients/TrustedIssuers at this layer).

Controller wiring — cmd/thv-operator/pkg/controllerutil/authserver.go

  • New buildSPIFFETrustDomainRunConfigs([]mcpv1beta1.SPIFFETrustDomainConfig) []authserver.SPIFFETrustDomainRunConfig
    and buildSPIFFEClientAuthRunConfigs([]mcpv1beta1.SPIFFEClientConfig) []authserver.SPIFFEClientAuthRunConfig
    (pure field copy + slice clone, no error — same shape as
    buildTrustedIssuerRunConfigs at line 269, since nothing here needs secret
    resolution).
  • buildInboundGrantsRunConfig (line 824): set
    grants.SPIFFEClientAuth = buildSPIFFEClientAuthRunConfigs(config.SPIFFEClientAuth)
    as a sibling assignment alongside the existing TokenExchange/JWTBearer
    blocks.
  • BuildAuthServerRunConfig (line 889): set
    config.SPIFFETrustDomains = buildSPIFFETrustDomainRunConfigs(authConfig.SPIFFETrustDomains)
    next to the existing config.InboundGrants = inboundGrants assignment.
  • validateDelegateClientsAndTrustedIssuers (line 1034): extend the early-
    return guard to also check len(config.SPIFFETrustDomains) == 0, and add
    SPIFFETrustDomains: config.SPIFFETrustDomains to validationConfig so
    RunConfig.Validate() catches SPIFFE misconfiguration (including the
    "not yet enforced" gate) as a reconcile error, not a pod crash loop —
    consistent with why this function exists at all.

Tests

  • Port the old commit's CEL test matrix
    (spiffe_cel_test.go, ~30 cases: paired-config, duplicate name/trustDomain/
    clientId/principalPattern, unreferenced/unknown trust-domain-ref,
    methods-not-a-subset, principal/trust-domain-segment mismatch, reserved
    clientId prefixes, trust-domain string edge cases, principal path-segment
    edge cases) rewritten against the new field path
    (inboundGrants.spiffeClientAuth, not .tokenExchange.spiffeClients) and
    extended for BundleSource's discriminated union and the new Resources
    field.
  • One reconcile-level test asserting a valid, non-empty SPIFFE CRD config is
    admitted by CEL but produces a terminal InvalidEmbeddedAuthServerConfigError
    reconcile failure (proving the "not yet enforced" gate is reachable end-to-
    end through the CRD path, not just the RunConfig-file path already tested
    in pkg/authserver).
  • Unit tests for the two new converter functions and
    validateSPIFFETrustConfig, following this file's existing table-driven
    convention.

Verification

  1. task operator-manifests then task operator-generate — confirm CRD
    YAML and deepcopy regenerate cleanly (watch for the same missing-JSON-tag
    class of controller-gen failure hit earlier in this session).
  2. task lint-fix — 0 issues.
  3. task test — full suite green, including the new CEL/reconcile tests
    (these need envtest/a real apiserver — same suite that already runs
    cmd/thv-operator/test-integration/mcp-external-auth/*_cel_test.go).
  4. Manually apply a VirtualMCPServer with valid SPIFFE fields against a
    local kind cluster (or envtest) and confirm: CRD admission succeeds, then
    the reconciler surfaces the expected "not yet enforced" terminal
    condition — not a crash loop, not silent success.

Delegation

Hand off implementation to a fresh agent (this is CRD/operator work spanning
Go API types, CEL, and controller wiring — kubernetes-expert or a fresh
go-expert-developer), then run an adversarial Opus review pass before
committing, matching this session's established mechanism for anything
security/protocol-relevant. Land as spiffe-integration-split3-6 on top of
the already-verified spiffe-integration-split3-5.

Special notes for reviewers

This PR was implemented by a fresh agent against the approved plan above, then went through one full adversarial review round: the reviewer found two real bugs (an admission-time Go check that unconditionally rejected any resources/custom-scopes entry due to a bad nil-handling assumption in the plan itself, and missing task crdref-gen output) plus an untested design deviation — all fixed and re-verified before this PR was opened. Stacked on #6499.

@github-actions github-actions Bot added the size/XL Extra large PR: 1000+ lines changed label Sep 3, 2026
@codecov

codecov Bot commented Sep 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.95960% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.40%. Comparing base (d634568) to head (e44e5c8).

Files with missing lines Patch % Lines
...perator/api/v1beta1/mcpexternalauthconfig_types.go 95.12% 2 Missing ⚠️
pkg/authserver/spiffe_trust.go 86.66% 2 Missing ⚠️
Additional details and impacted files
@@                       Coverage Diff                       @@
##           spiffe-integration-split3-5    #6500      +/-   ##
===============================================================
+ Coverage                        78.36%   78.40%   +0.04%     
===============================================================
  Files                              776      776              
  Lines                            76175    76253      +78     
===============================================================
+ Hits                             59693    59788      +95     
+ Misses                           16477    16460      -17     
  Partials                             5        5              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@JAORMX JAORMX left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CRD conversion and generated artifacts are largely aligned, but the panel found material validation/lifecycle gaps:

  1. principalPattern rejects ~ in path segments (cmd/thv-operator/api/v1beta1/mcpexternalauthconfig_types.go:731-738), although it is valid in SPIFFE URI path segments. This makes the served CRD schema narrower than the authoritative runtime parser. Please accept the full permitted segment grammar (including ~) and add concrete/wildcard admission coverage.

  2. Context-independent SPIFFE errors can still lead MCPExternalAuthConfig to report Valid=True: bundle endpoint URLs are not structurally validated and overlapping principal policies are only rejected later by runtime validation (cmd/thv-operator/api/v1beta1/mcpexternalauthconfig_types.go:647-653, 890-891, 2366-2373). Validate these at admission or in the config controller; defer only checks that genuinely require workload-derived allowlists.

  3. The multi-issuer validator owns background jwk.Cache workers but neither it nor the auth server closes them; constructor error paths also leave earlier caches running (pkg/authserver/server/tokenexchange/multi_issuer_validator.go:415-445,549; pkg/authserver/server_impl.go:396-400). Add lifecycle cleanup on server close and on partial construction failure, with tests.

The red Go Vulnerability Check is inherited from the unchanged dependency in the stacked base and appears unrelated.

@jhrozek
jhrozek force-pushed the spiffe-integration-split3-5 branch from 5834d9c to 6dbb23e Compare September 3, 2026 10:04
@jhrozek
jhrozek force-pushed the spiffe-integration-split3-6 branch from b758757 to f57d37e Compare September 3, 2026 10:06
@github-actions github-actions Bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Sep 3, 2026
@jhrozek
jhrozek force-pushed the spiffe-integration-split3-5 branch from 6dbb23e to e038ceb Compare September 3, 2026 13:03
@jhrozek
jhrozek force-pushed the spiffe-integration-split3-6 branch from f57d37e to c5e6f6e Compare September 3, 2026 13:05
@github-actions github-actions Bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Sep 3, 2026

@JAORMX JAORMX left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at c5e6f6e6654575c784b132321caf72c543d9dd3a: none of the three prior blockers are resolved, and the same inline-configuration parity gap exists here.

  • CRD principalPattern still rejects valid SPIFFE ~ path segments (cmd/thv-operator/api/v1beta1/mcpexternalauthconfig_types.go:731-738).
  • Malformed bundle endpoints and overlapping principal patterns still bypass validation on the owning config CRD and can be reported Valid=True (mcpexternalauthconfig_types.go:647-653,883-893; controllers/mcpexternalauthconfig_controller.go:106-176).
  • JWK cache workers still have no cleanup on server shutdown or partial validator construction (pkg/authserver/server/tokenexchange/multi_issuer_validator.go:433-470,557-578; pkg/authserver/server_impl.go:375-400).
  • VirtualMCPServer also still rejects advertised SPIFFE-only inline configuration because its guard ignores inboundGrants.spiffeClientAuth (cmd/thv-operator/controllers/virtualmcpserver_controller.go:670-685).

CI is green; no local tests were run.

@JAORMX JAORMX left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested:

  • cmd/thv-operator/api/v1beta1/mcpexternalauthconfig_types.go:737: principalPattern excludes ~, which the authoritative SPIFFE parser permits in path segments. Permit it and add admission coverage.
  • cmd/thv-operator/api/v1beta1/mcpexternalauthconfig_types.go:649-653,2366-2373 and cmd/thv-operator/pkg/controllerutil/authserver.go:1022-1024: malformed bundle endpoints and overlapping principal patterns are published Valid=True until a consuming workload fails. Validate context-independent invariants at the owning config boundary.
  • cmd/thv-operator/controllers/virtualmcpserver_controller.go:670: include inboundGrants.spiffeClientAuth in inline auth-server validation; the shared CRD currently admits a SPIFFE-only shape that this guard rejects.

@jhrozek
jhrozek force-pushed the spiffe-integration-split3-5 branch from e038ceb to ae26bc4 Compare September 3, 2026 14:14
@jhrozek
jhrozek force-pushed the spiffe-integration-split3-6 branch from c5e6f6e to ff2fc59 Compare September 3, 2026 14:15
@github-actions github-actions Bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Sep 3, 2026

@JAORMX JAORMX left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested:

  • cmd/thv-operator/api/v1beta1/mcpexternalauthconfig_types.go:737: principalPattern still rejects ~, which the authoritative SPIFFE parser permits in path segments. Permit it and add concrete and wildcard admission coverage.
  • cmd/thv-operator/api/v1beta1/mcpexternalauthconfig_types.go:649-653,890-891 and cmd/thv-operator/controllers/mcpexternalauthconfig_controller.go:106-176: malformed bundle endpoints and overlapping principal patterns can be published Valid=True at the owning config boundary and only fail when a consuming workload validates the runtime configuration. Validate these context-independent invariants at admission or in the config controller.
  • cmd/thv-operator/controllers/virtualmcpserver_controller.go:670-685: include inboundGrants.spiffeClientAuth in inline auth-server validation; a SPIFFE-only inline shape admitted by the shared CRD is still rejected here.

All current CI checks are green; no local tests were run.

@jhrozek
jhrozek force-pushed the spiffe-integration-split3-5 branch from ae26bc4 to ce2447f Compare September 3, 2026 16:16
@jhrozek
jhrozek force-pushed the spiffe-integration-split3-6 branch from ff2fc59 to 3b10dc8 Compare September 3, 2026 16:24

@JAORMX JAORMX left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at 3b10dc81849b9aae787cc77b600cc5714f2c9daf:

  • cmd/thv-operator/api/v1beta1/mcpexternalauthconfig_types.go:737 still rejects ~ in SPIFFE principal path segments. ~ is valid in SPIFFE URI path segments, so the served CRD schema and runtime pattern validation are both narrower than the authoritative grammar. Permit it and add concrete plus terminal-wildcard admission coverage for both MCPExternalAuthConfig and VirtualMCPServer.

The previous ~ blocker remains unresolved.

@jhrozek
jhrozek force-pushed the spiffe-integration-split3-5 branch from ce2447f to 1d2f7a3 Compare September 3, 2026 18:42
@jhrozek
jhrozek force-pushed the spiffe-integration-split3-6 branch from 3b10dc8 to 5545daa Compare September 3, 2026 18:46
@github-actions github-actions Bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Sep 3, 2026
@jhrozek
jhrozek force-pushed the spiffe-integration-split3-5 branch from 1d2f7a3 to 5314ea2 Compare September 4, 2026 05:44
@jhrozek
jhrozek force-pushed the spiffe-integration-split3-6 branch from 5545daa to a314f80 Compare September 4, 2026 05:53
@github-actions github-actions Bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Sep 4, 2026

@JAORMX JAORMX left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deep re-review at 5545daa9c87e7ef9b53f893141aad1f65f8b6434 confirms the prior SPIFFE grammar blocker and finds another correctness gap:

  • High: SPIFFE resources are independently modeled but discarded while constructing static OAuth clients (pkg/authserver/spiffe_association_registry.go:56-58; pkg/authserver/server/registration/spiffe_client.go:31-44,69-72). Both RFC 8693 audience and RFC 8707 resource are then authorized via GetAudience() (pkg/authserver/server/tokenexchange/handler.go:810-816,820-851). For disjoint audiences: [A] and resources: [R], resource=R is denied and resource=A is incorrectly permitted. Preserve and authorize resources separately; add four-way resource/audience tests.
  • Medium: durable SPIFFE placeholder reservations are never removed if an association disappears. spiffe_decorator.go:60-83 reconciles desired IDs only and neither decorator nor memory/Redis storage has a removal path (pkg/authserver/storage/spiffe_decorator.go:25-47, memory.go:528-546, redis.go:542-562). A removed/renamed SPIFFE ID stays unavailable indefinitely. Reconcile removals only for placeholders owned by this auth-server configuration; prove removed IDs become usable while active ones stay protected.

The prior valid-~ principal-pattern blocker remains.

@jhrozek

jhrozek commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

Pushing back on the ~ finding rather than fixing it — checked this against the actual runtime parser we use.

pkg/authserver/spiffe_trust.go validates principals via spiffeid.FromString from github.com/spiffe/go-spiffe/v2 (pinned at v2.7.0 in go.mod). That library's path-segment charset is [a-zA-Z0-9._-] by default; ~ is only accepted if the module is built with the spiffeid_charset_backcompat build tag, which doesn't appear anywhere in this repo (checked go.mod, Taskfile, CI workflows, build tags in our own code). With the default build, spiffeid.FromString("spiffe://example.org/ns/default/agent~1") returns an error in go-spiffe itself.

So the CRD regex at mcpexternalauthconfig_types.go:737 and the runtime parser agree — both reject ~. Widening the CRD pattern to admit it would mean the CRD accepts SPIFFE IDs that NewSPIFFETrustConfig then rejects at reconcile time, which is worse than what we have now. The two tests you'd want us to remove (spiffe_cel_test.go:289,293) are asserting the correct behavior for this dependency.

Happy to be wrong here — if there's a reason to vendor go-spiffe with the backcompat tag, or if the concern is about the SPIFFE-ID spec's normative text rather than this specific library's default build, let me know and I'll take another look. Otherwise I'd like to close this one out as-is rather than keep carrying it forward.

@jhrozek
jhrozek force-pushed the spiffe-integration-split3-5 branch from 5314ea2 to 291104f Compare September 4, 2026 08:07
@jhrozek
jhrozek force-pushed the spiffe-integration-split3-6 branch from a314f80 to 4565c8d Compare September 4, 2026 08:09
@github-actions github-actions Bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Sep 4, 2026
@jhrozek
jhrozek force-pushed the spiffe-integration-split3-5 branch from 291104f to 40a2370 Compare September 4, 2026 12:31
@jhrozek
jhrozek force-pushed the spiffe-integration-split3-6 branch from 4565c8d to 66edc12 Compare September 4, 2026 12:34
@github-actions github-actions Bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Sep 4, 2026

@JAORMX JAORMX left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at 66edc122efed74572d9a4d9784a1ef83e5455afe: malformed bundle endpoints and overlapping principal patterns are now validated at the owning configuration boundary, the base now preserves/enforces RFC 8707 resources, and the JWK cache lifecycle issue was fixed on main. Those prior concerns are resolved.

One blocker remains unchanged:

  • cmd/thv-operator/api/v1beta1/mcpexternalauthconfig_types.go:731-738 rejects ~ in principalPattern path segments, although ~ is valid in a SPIFFE ID path segment. This makes the served v1beta1 schema narrower than the authoritative SPIFFE grammar. Permit it in both the CRD marker and runtime pattern validation, and add concrete plus terminal-wildcard admission coverage for both MCPExternalAuthConfig and VirtualMCPServer.

The updated head is a stack rebuild; this layer did not change relative to the previously reviewed implementation. All checks are green.

@jhrozek
jhrozek force-pushed the spiffe-integration-split3-5 branch from 40a2370 to d634568 Compare September 5, 2026 13:45
The SPIFFE client-auth epic needs a place to configure SPIFFE
association policy without inventing a parallel trust/grant path next
to the existing delegate-client and trusted-issuer configuration. As
more inbound grant families (RFC 8693 token exchange, RFC 7523
JWT-bearer, SPIFFE) accumulate, they need one canonical surface to
configure and reason about instead of three independent ones, without
breaking deployments that already rely on the legacy fields.

Add pkg/authserver/inbound_grants.go with NormalizeInboundGrants, which
reconciles a new canonical RunConfig.InboundGrants surface (per-family
token_exchange/jwt_bearer sub-configs whose issuer_policies reference a
trusted_issuers entry by name) against the legacy top-level
delegate_clients and the RFC 8693/7523 fields embedded directly on
trusted_issuers. Legacy and canonical configuration for the same grant
family are mutually exclusive and rejected at validation time; the two
families are otherwise independent, and omitting inbound_grants
entirely preserves released behavior. Thread the normalized result
through RunConfig.Validate, the embedded-auth-server runner, and
buildProvider/discovery, adding a DisableTokenExchange capability so
RFC 8693 registration and discovery advertisement can be turned off
together and can't drift out of sync. Add TrustedIssuer.Name so
canonical issuer_policies can reference an issuer without duplicating
its fields.

SPIFFE client authentication (InboundGrants.SPIFFEClientAuth, defined
in the previous commit) is deliberately kept a sibling of TokenExchange
and JWTBearer here, not nested under either: SPIFFE authenticates a
client, it does not by itself grant it anything, so making it subordinate
to RFC 8693 enablement would mean disabling token exchange silently drops
every SPIFFE association, and every SPIFFE-authenticated client would be
implicitly token-exchange-capable. It is validated and wired directly
from RunConfig.InboundGrants in RunConfig.Validate/embeddedauthserver.go,
independent of this file's legacy/canonical projection, so authentication
method and grant-family enablement stay separately configurable.

Update docs/arch/17-token-exchange-delegation.md for the new
inbound_grants shape and the now-conditional token-exchange discovery
advertisement, and add a runner-level test proving the canonical
delegate-client, SPIFFE-client, and jwt_bearer paths reach a running
server (the existing tests only covered normalization in isolation).

SPIFFE client-auth associations always require the token-exchange
grant (the only grant type they may declare), independent of the
legacy/canonical token-exchange projection above: NormalizeInboundGrants
now sets Capabilities.TokenExchange true whenever
InboundGrants.SPIFFEClientAuth is non-empty, so a SPIFFE-only
configuration cannot leave it false and silently disable the RFC 8693
grant handler server-wide -- which would reject every SPIFFE client's
own token requests before authentication is even checked. Guarded by a
regression test in this package (not just the runner-level test above)
since the equivalent fix was previously lost during a rebase when its
only coverage lived one package away.

DCR (RFC 7591 /oauth/register) now rejects a registration whose
effective grant types include token-exchange when it is disabled
server-wide, instead of accepting the client and only failing later,
confusingly, at /oauth/token. The check runs on the post-defaulting
grant types validateGrantTypes already computes (a private_key_jwt
client with an empty grant_types is implicitly token-exchange-only),
so it catches both the explicit and implicit cases the same way scope
validation already gates DCR on ScopesSupported.

Corrected two stale doc references caught in review: the SPIFFE
client-policy field path (inbound_grants.spiffe_client_auth, not
nested under token_exchange) and the JWT-bearer legacy/canonical
conflict wording (family-wide across all issuers, not per-issuer).

Refs #6200

Signed-off-by: Jakub Hrozek <jakub@stacklok.com>
The SPIFFE client-auth epic needs a place to configure SPIFFE
association policy without inventing a parallel trust/grant path next
to the existing delegate-client and trusted-issuer configuration. As
more inbound grant families (RFC 8693 token exchange, RFC 7523
JWT-bearer, SPIFFE) accumulate, they need one canonical surface to
configure and reason about instead of three independent ones, without
breaking deployments that already rely on the legacy fields.

Add pkg/authserver/inbound_grants.go with NormalizeInboundGrants, which
reconciles a new canonical RunConfig.InboundGrants surface (per-family
token_exchange/jwt_bearer sub-configs whose issuer_policies reference a
trusted_issuers entry by name) against the legacy top-level
delegate_clients and the RFC 8693/7523 fields embedded directly on
trusted_issuers. Legacy and canonical configuration for the same grant
family are mutually exclusive and rejected at validation time; the two
families are otherwise independent, and omitting inbound_grants
entirely preserves released behavior. Thread the normalized result
through RunConfig.Validate, the embedded-auth-server runner, and
buildProvider/discovery, adding a DisableTokenExchange capability so
RFC 8693 registration and discovery advertisement can be turned off
together and can't drift out of sync. Add TrustedIssuer.Name so
canonical issuer_policies can reference an issuer without duplicating
its fields.

SPIFFE client authentication (InboundGrants.SPIFFEClientAuth, defined
in the previous commit) is deliberately kept a sibling of TokenExchange
and JWTBearer here, not nested under either: SPIFFE authenticates a
client, it does not by itself grant it anything, so making it subordinate
to RFC 8693 enablement would mean disabling token exchange silently drops
every SPIFFE association, and every SPIFFE-authenticated client would be
implicitly token-exchange-capable. It is validated and wired directly
from RunConfig.InboundGrants in RunConfig.Validate/embeddedauthserver.go,
independent of this file's legacy/canonical projection, so authentication
method and grant-family enablement stay separately configurable.

Update docs/arch/17-token-exchange-delegation.md for the new
inbound_grants shape and the now-conditional token-exchange discovery
advertisement, and add a runner-level test proving the canonical
delegate-client, SPIFFE-client, and jwt_bearer paths reach a running
server (the existing tests only covered normalization in isolation).

SPIFFE client-auth associations always require the token-exchange
grant (the only grant type they may declare), independent of the
legacy/canonical token-exchange projection above: NormalizeInboundGrants
now sets Capabilities.TokenExchange true whenever
InboundGrants.SPIFFEClientAuth is non-empty, so a SPIFFE-only
configuration cannot leave it false and silently disable the RFC 8693
grant handler server-wide -- which would reject every SPIFFE client's
own token requests before authentication is even checked. Guarded by a
regression test in this package (not just the runner-level test above)
since the equivalent fix was previously lost during a rebase when its
only coverage lived one package away.

DCR (RFC 7591 /oauth/register) now rejects a registration whose
effective grant types include token-exchange when it is disabled
server-wide, instead of accepting the client and only failing later,
confusingly, at /oauth/token. The check runs on the post-defaulting
grant types validateGrantTypes already computes (a private_key_jwt
client with an empty grant_types is implicitly token-exchange-only),
so it catches both the explicit and implicit cases the same way scope
validation already gates DCR on ScopesSupported.

Corrected two stale doc references caught in review: the SPIFFE
client-policy field path (inbound_grants.spiffe_client_auth, not
nested under token_exchange) and the JWT-bearer legacy/canonical
conflict wording (family-wide across all issuers, not per-issuer).

Refs #6200

Signed-off-by: Jakub Hrozek <jakub@stacklok.com>
The SPIFFE client-auth epic needs a place to configure SPIFFE
association policy without inventing a parallel trust/grant path next
to the existing delegate-client and trusted-issuer configuration. As
more inbound grant families (RFC 8693 token exchange, RFC 7523
JWT-bearer, SPIFFE) accumulate, they need one canonical surface to
configure and reason about instead of three independent ones, without
breaking deployments that already rely on the legacy fields.

Add pkg/authserver/inbound_grants.go with NormalizeInboundGrants, which
reconciles a new canonical RunConfig.InboundGrants surface (per-family
token_exchange/jwt_bearer sub-configs whose issuer_policies reference a
trusted_issuers entry by name) against the legacy top-level
delegate_clients and the RFC 8693/7523 fields embedded directly on
trusted_issuers. Legacy and canonical configuration for the same grant
family are mutually exclusive and rejected at validation time; the two
families are otherwise independent, and omitting inbound_grants
entirely preserves released behavior. Thread the normalized result
through RunConfig.Validate, the embedded-auth-server runner, and
buildProvider/discovery, adding a DisableTokenExchange capability so
RFC 8693 registration and discovery advertisement can be turned off
together and can't drift out of sync. Add TrustedIssuer.Name so
canonical issuer_policies can reference an issuer without duplicating
its fields.

SPIFFE client authentication (InboundGrants.SPIFFEClientAuth, defined
in the previous commit) is deliberately kept a sibling of TokenExchange
and JWTBearer here, not nested under either: SPIFFE authenticates a
client, it does not by itself grant it anything, so making it subordinate
to RFC 8693 enablement would mean disabling token exchange silently drops
every SPIFFE association, and every SPIFFE-authenticated client would be
implicitly token-exchange-capable. It is validated and wired directly
from RunConfig.InboundGrants in RunConfig.Validate/embeddedauthserver.go,
independent of this file's legacy/canonical projection, so authentication
method and grant-family enablement stay separately configurable.

Update docs/arch/17-token-exchange-delegation.md for the new
inbound_grants shape and the now-conditional token-exchange discovery
advertisement, and add a runner-level test proving the canonical
delegate-client, SPIFFE-client, and jwt_bearer paths reach a running
server (the existing tests only covered normalization in isolation).

SPIFFE client-auth associations always require the token-exchange
grant (the only grant type they may declare), independent of the
legacy/canonical token-exchange projection above: NormalizeInboundGrants
now sets Capabilities.TokenExchange true whenever
InboundGrants.SPIFFEClientAuth is non-empty, so a SPIFFE-only
configuration cannot leave it false and silently disable the RFC 8693
grant handler server-wide -- which would reject every SPIFFE client's
own token requests before authentication is even checked. Guarded by a
regression test in this package (not just the runner-level test above)
since the equivalent fix was previously lost during a rebase when its
only coverage lived one package away.

DCR (RFC 7591 /oauth/register) now rejects a registration whose
effective grant types include token-exchange when it is disabled
server-wide, instead of accepting the client and only failing later,
confusingly, at /oauth/token. The check runs on the post-defaulting
grant types validateGrantTypes already computes (a private_key_jwt
client with an empty grant_types is implicitly token-exchange-only),
so it catches both the explicit and implicit cases the same way scope
validation already gates DCR on ScopesSupported.

Corrected two stale doc references caught in review: the SPIFFE
client-policy field path (inbound_grants.spiffe_client_auth, not
nested under token_exchange) and the JWT-bearer legacy/canonical
conflict wording (family-wide across all issuers, not per-issuer).

Refs #6200

Signed-off-by: Jakub Hrozek <jakub@stacklok.com>
The SPIFFE client-auth epic needs a place to configure SPIFFE
association policy without inventing a parallel trust/grant path next
to the existing delegate-client and trusted-issuer configuration. As
more inbound grant families (RFC 8693 token exchange, RFC 7523
JWT-bearer, SPIFFE) accumulate, they need one canonical surface to
configure and reason about instead of three independent ones, without
breaking deployments that already rely on the legacy fields.

Add pkg/authserver/inbound_grants.go with NormalizeInboundGrants, which
reconciles a new canonical RunConfig.InboundGrants surface (per-family
token_exchange/jwt_bearer sub-configs whose issuer_policies reference a
trusted_issuers entry by name) against the legacy top-level
delegate_clients and the RFC 8693/7523 fields embedded directly on
trusted_issuers. Legacy and canonical configuration for the same grant
family are mutually exclusive and rejected at validation time; the two
families are otherwise independent, and omitting inbound_grants
entirely preserves released behavior. Thread the normalized result
through RunConfig.Validate, the embedded-auth-server runner, and
buildProvider/discovery, adding a DisableTokenExchange capability so
RFC 8693 registration and discovery advertisement can be turned off
together and can't drift out of sync. Add TrustedIssuer.Name so
canonical issuer_policies can reference an issuer without duplicating
its fields.

SPIFFE client authentication (InboundGrants.SPIFFEClientAuth, defined
in the previous commit) is deliberately kept a sibling of TokenExchange
and JWTBearer here, not nested under either: SPIFFE authenticates a
client, it does not by itself grant it anything, so making it subordinate
to RFC 8693 enablement would mean disabling token exchange silently drops
every SPIFFE association, and every SPIFFE-authenticated client would be
implicitly token-exchange-capable. It is validated and wired directly
from RunConfig.InboundGrants in RunConfig.Validate/embeddedauthserver.go,
independent of this file's legacy/canonical projection, so authentication
method and grant-family enablement stay separately configurable.

Update docs/arch/17-token-exchange-delegation.md for the new
inbound_grants shape and the now-conditional token-exchange discovery
advertisement, and add a runner-level test proving the canonical
delegate-client, SPIFFE-client, and jwt_bearer paths reach a running
server (the existing tests only covered normalization in isolation).

SPIFFE client-auth associations always require the token-exchange
grant (the only grant type they may declare), independent of the
legacy/canonical token-exchange projection above: NormalizeInboundGrants
now sets Capabilities.TokenExchange true whenever
InboundGrants.SPIFFEClientAuth is non-empty, so a SPIFFE-only
configuration cannot leave it false and silently disable the RFC 8693
grant handler server-wide -- which would reject every SPIFFE client's
own token requests before authentication is even checked. Guarded by a
regression test in this package (not just the runner-level test above)
since the equivalent fix was previously lost during a rebase when its
only coverage lived one package away.

DCR (RFC 7591 /oauth/register) now rejects a registration whose
effective grant types include token-exchange when it is disabled
server-wide, instead of accepting the client and only failing later,
confusingly, at /oauth/token. The check runs on the post-defaulting
grant types validateGrantTypes already computes (a private_key_jwt
client with an empty grant_types is implicitly token-exchange-only),
so it catches both the explicit and implicit cases the same way scope
validation already gates DCR on ScopesSupported.

Corrected two stale doc references caught in review: the SPIFFE
client-policy field path (inbound_grants.spiffe_client_auth, not
nested under token_exchange) and the JWT-bearer legacy/canonical
conflict wording (family-wide across all issuers, not per-issuer).

Refs #6200

Signed-off-by: Jakub Hrozek <jakub@stacklok.com>
The SPIFFE client-authentication runtime model (trust domains + static
workload-to-client associations) already exists in pkg/authserver, but
was only reachable by hand-authoring an auth-server RunConfig file --
there was no way to declare it through VirtualMCPServer or
MCPExternalAuthConfig.

Add SPIFFETrustDomains and InboundGrants.SPIFFEClientAuth to the shared
EmbeddedAuthServerConfig CRD type, with CEL admission validation for
everything derivable from the object's own spec (paired configuration,
no duplicate names/trust domains/client IDs/principal patterns, full
cross-referencing between domains and clients, method-subset
enforcement, principal/trust-domain consistency, and client-ID
hygiene). Wire the new fields through to authserver.RunConfig via the
same converter pattern used for DelegateClients/TrustedIssuers, and
extend the existing reconcile-time revalidation
(validateDelegateClientsAndTrustedIssuers) to cover SPIFFE trust
domains too.

Cross-field checks that need reconcile-time-derived values
(AllowedAudiences, ScopesSupported -- neither is CRD-exposed) are left
to that reconcile-time path rather than a premature admission-time Go
check: an earlier attempt at that check passed nil for both, which the
runtime validator treats as "validate against nothing" rather than
"skip", silently rejecting any resources or custom scopes entry.

A syntactically valid SPIFFE config is admitted by CEL but still fails
reconciliation with a terminal error until RunConfig's
"not yet enforced" placeholder gate is lifted by a future PR -- no
live SVID verification exists yet, so this is expected, not a
regression.

Signed-off-by: Jakub Hrozek <jakub@stacklok.com>
@jhrozek
jhrozek force-pushed the spiffe-integration-split3-6 branch from 66edc12 to e44e5c8 Compare September 5, 2026 13:48
@github-actions github-actions Bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Sep 5, 2026

@JAORMX JAORMX left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at e44e5c8 (reviewed only d634568..e44e5c8):

Resolved: bundle endpoints and overlapping principals are validated at the owning configuration boundary; SPIFFE-only inline config is accepted; resources are preserved; generated artifacts are synchronized.

Remaining blockers:

  • cmd/thv-operator/api/v1beta1/mcpexternalauthconfig_types.go:731-738; pkg/authserver/spiffe_trust.go:663-690: principalPattern still rejects ~, which is valid in SPIFFE path segments. Permit it in CRD and runtime validation, regenerate output, and add concrete plus terminal-wildcard coverage for both CRDs.
  • cmd/thv-operator/api/v1beta1/mcpexternalauthconfig_types.go:756-780,2368-2374: context-independent malformed/empty audiences, scopes, and resource URI shapes can be admitted and reported Valid=True because all permission validation is deferred. Keep derived allowlist checks at reconciliation, but validate independent shape constraints at the owning boundary.

Exact-head CI is green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XL Extra large PR: 1000+ lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants