feat(auth): add SPIFFE supervisor authentication#1414
Draft
TaylorMutch wants to merge 12 commits into
Draft
Conversation
Replaces the hard-coded sandbox-method / dual-auth / Bearer branches in
AuthGrpcRouter with a pluggable Authenticator chain that produces a
Principal::{User, Sandbox, Anonymous}. The principal is inserted into
request extensions for handler consumption.
PR-1 keeps the legacy metadata marker for sandbox principals so existing
handlers that read x-openshell-auth-source continue to work; the marker
is removed in the PR-3 wire break. The OidcAuthenticator wraps the
existing JwksCache::validate_token for User principals, and the
LegacySandboxMarkerAuthenticator preserves the pre-refactor path-based
behavior pending the gateway-minted JWT flow in PR 2/3.
Part of the per-sandbox identity series that closes #1354.
Adds the gateway-side infrastructure for per-sandbox identity tokens (the PR-2 step of the series resolving #1354): - New Ed25519 keypair generated by `certgen` alongside the existing PKI. Local mode writes `<dir>/jwt/{signing.pem,public.pem,kid}`; K8s mode creates an Opaque `<release>-jwt-keys` Secret. - `SandboxJwtIssuer` mints tokens with EdDSA-signed claims (SPIFFE-shaped `sub`, denormalised `sandbox_id`, 24h default TTL, `jti` for revocation). - `SandboxJwtAuthenticator` validates tokens through the Authenticator chain and yields `Principal::Sandbox(BootstrapJwt {..})`. Tokens with a different `kid` fall through so non-matching Bearer headers reach the OIDC authenticator unchanged. - `K8sServiceAccountAuthenticator` is path-scoped to `IssueSandboxToken`; consumes a projected SA token and produces a `K8sServiceAccount` sandbox principal that the new `IssueSandboxToken` handler exchanges for a fresh gateway JWT. - In-memory `RevocationSet` with TTL pruning, ready for the PR-3 delete-side hook and PR-5 refresh. - Helm chart mounts the JWT secret on the gateway pod and wires `[openshell.gateway.gateway_jwt]` into the rendered TOML. PR 2 is additive: no driver yet writes a sandbox token, no supervisor yet presents a Bearer JWT. PR 3 wires the consumer ends and removes the legacy path-based sandbox marker.
Switches every sandbox-to-gateway gRPC call from "path-based mTLS-only trust" to "Authorization: Bearer <gateway-minted-JWT>" presented by the sandbox supervisor. Closes the trust-boundary half of issue #1354; the per-handler sandbox_id equality check follows in PR 4. Sandbox side: - crates/openshell-sandbox/src/grpc_client.rs gains an AuthInterceptor that injects the Bearer header on every outbound RPC. The token is resolved at startup from one of three sources, in order: 1. OPENSHELL_SANDBOX_TOKEN (env, test harnesses) 2. OPENSHELL_SANDBOX_TOKEN_FILE (Docker/Podman/VM drivers) 3. OPENSHELL_K8S_SA_TOKEN_FILE (K8s driver — projected SA token exchanged for a gateway JWT via IssueSandboxToken) Gateway side: - handle_create_sandbox mints a gateway JWT and passes it through the compute layer to DriverSandboxSpec.sandbox_token. K8s sandboxes ignore the field; Docker and Podman drivers inject it as OPENSHELL_SANDBOX_TOKEN in the container env. - Removes the path-based SANDBOX_METHODS / DUAL_AUTH_METHODS branches and the x-openshell-auth-source metadata marker. The AuthGrpcRouter chain is now uniform: K8s SA -> SandboxJwt -> OIDC, all extension-based. - Removes LegacySandboxMarkerAuthenticator and the SandboxIdentitySource:: LegacyMarker variant. Handlers read Principal::Sandbox directly from request extensions. Kubernetes driver: - Sandbox pods gain a projected ServiceAccount token volume mounted at /var/run/secrets/openshell/token (audience openshell-gateway, 1h TTL, kubelet auto-rotates). - Each pod is annotated with openshell.io/sandbox-id; the gateway resolves the SA token claim's pod uid back to a sandbox id via this annotation. - Helm Role grants the gateway pods:get in the sandbox namespace. No ClusterRoleBinding to system:auth-delegator — the gateway validates SA tokens against the apiserver's anonymous JWKS endpoint instead of via TokenReview, so no cluster-scoped privilege is required. The full JWKS verifier + pod-annotation lookup lands in the follow-up that brings the K8s helm-dev demo end-to-end; PR 3 exercises the wire break with Docker/Podman as the working drivers.
ProcessHandle::spawn_impl previously inherited the supervisor's full environment when starting the sandbox entrypoint, then drop_privileges() demoted the child to the sandbox user. The combination meant a later process running as the sandbox user (e.g. an SSH-spawned shell) could read /proc/<entrypoint_pid>/environ and recover the gateway-minted JWT. Explicitly env_remove the three sandbox-token env vars before exec so the entrypoint child carries none of the supervisor's identity material. SSH session shells already use env_clear() in apply_child_env, so this plugs the only remaining inheritance path. Related to #1354 (per-sandbox identity series, PR 3 follow-up).
Adds the IDOR guard that closes the second half of the per-sandbox identity series. Every sandbox-class handler now verifies that the calling Principal::Sandbox.sandbox_id matches the canonical UUID the request body operates on. User principals bypass the check because RBAC was their gate at the router layer; anonymous callers are rejected outright. New module crates/openshell-server/src/auth/guard.rs exposes ensure_sandbox_scope / enforce_sandbox_scope. Applied at the top of: - handle_get_sandbox_config (id-keyed) - handle_get_sandbox_provider_environment (id-keyed) - handle_report_policy_status (id-keyed) - handle_push_sandbox_logs (id-keyed, first frame only — principal is stable across the stream) - handle_submit_policy_analysis (name-keyed: resolve to id, then check) - handle_get_draft_policy (name-keyed) - handle_update_config (dual-auth: enforce only when Principal::Sandbox; CLI / TUI user paths are unaffected) - handle_get_inference_bundle (no sandbox_id in body; accept any authenticated principal, reject anonymous) Existing policy.rs tests are updated to wrap their requests with a test-helper user principal so the new guard treats them as CLI calls; six new tests cover the cross-sandbox-denied / same-sandbox-allowed / user-bypasses-guard matrix.
Adds the rotation half of the per-sandbox identity series. Sandboxes holding a valid gateway-minted JWT can swap it for a fresh one without disruption; the old jti is revoked server-side before the new token is handed back, so a leaked token is unusable as soon as the rotation completes. Server side: - proto/openshell.proto gains RefreshSandboxToken plus empty request / token+expires_at_ms response messages. - handle_refresh_sandbox_token requires Principal::Sandbox with a BootstrapJwt source (K8s-SA principals are routed to IssueSandboxToken for bootstrap; user principals are rejected). The handler mints the replacement token first, then adds the old jti to the in-memory RevocationSet — so a failed mint never strands the sandbox. Sandbox side: - AuthInterceptor now reads its Bearer header from a process-wide Arc<RwLock<AsciiMetadataValue>> slot, so a single in-place token rotation is visible to every cached client (CachedOpenShellClient, the supervisor session channel, log push, etc.). - connect_channel spawns a background refresh loop once per process that sleeps for ~80% of the token's remaining lifetime (clamped to 60s-12h, plus small deterministic jitter) and calls RefreshSandboxToken, updating the token slot on success. - New parse_jwt_exp_ms helper decodes the JWT payload without signature verification — the token's origin is already trusted via the acquisition flow. Tests: - 4 server-side handler tests (round-trip, user-principal rejected, K8s-SA-principal rejected, missing-issuer returns Unavailable) - 3 sandbox-side helper tests (parse-exp, 80%-of-TTL delay, 60s floor) All existing OpenShell test impls gain a refresh_sandbox_token stub.
The projected SA token kubelet writes to each sandbox pod was previously a hardcoded 3600s literal in the driver. Operators in tighter audit regimes want to dial it lower; very large clusters may want it slightly higher to absorb token-refresh churn. Wires `sa_token_ttl_secs` through three layers: - KubernetesComputeConfig gains the field (default 3600). The driver clamps to [600, 86400] via `effective_sa_token_ttl_secs()`: 600s is kubelet's enforced minimum, 24h is the cap (the token is consumed within seconds of pod start, so longer is almost always a misconfiguration). - The openshell-driver-kubernetes binary exposes `--sa-token-ttl-secs` / `OPENSHELL_K8S_SA_TOKEN_TTL_SECS`. - `[openshell.gateway].sa_token_ttl_secs` in the gateway TOML inherits into `[openshell.drivers.kubernetes]`, mirroring the `enable_user_namespaces` plumbing. - Helm: `server.sandboxJwt.k8sSaTokenTtlSecs` (default 3600) renders into the K8s driver block of the gateway config.
Replaces the LiveK8sResolver stub with a working validator. Sandbox pods
present their projected ServiceAccount token via Authorization: Bearer
on IssueSandboxToken; the gateway:
1. Decodes the JWT header and looks up the signing key.
2. On miss, fetches the apiserver's /.well-known/openid-configuration
discovery doc + /openid/v1/jwks via kube::Client and caches the keys.
3. Validates the token's signature (RS256), issuer, audience
(openshell-gateway), and expiry.
4. Reads `kubernetes.io.pod.{name,uid}` from the claims and GETs the
pod in the gateway's sandbox namespace.
5. Verifies the live pod's UID matches the token's UID (defense against
replayed tokens from recreated pods with the same name) and reads
the openshell.io/sandbox-id annotation to derive the sandbox UUID.
The gateway needs no system:auth-delegator ClusterRoleBinding — JWKS
validation is local, so the only K8s permission it consumes is the
namespace Role's `pods: get` grant. Discovery + JWKS reads ride the
gateway's existing kube::Client auth (system:service-account-issuer-
discovery is bound to system:authenticated in every supported K8s
distro).
ServerState gains an in-cluster detection path in run_server: when
KUBERNETES_SERVICE_HOST is set AND a sandbox JWT issuer is configured,
construct the resolver and wire it as state.k8s_sa_authenticator. The
existing K8sServiceAccountAuthenticator (path-scoped to
IssueSandboxToken) becomes functional.
Tests: JWKS path parsing covers absolute URL, relative path, query
string, and garbage rejection. End-to-end validation against a real
apiserver is exercised in the helm-dev demo.
Three regressions / inefficiencies surfaced while bringing the per-sandbox identity series up end-to-end in the local helm cluster: 1. CLI returned Unauthenticated against a no-OIDC dev gateway. PR 3 removed the pre-refactor "no OIDC = pass through" behavior; with only sandbox-side authenticators in the chain, plain user CLI calls hit Unauthenticated. Add a PermissiveUserAuthenticator that installs as a final fallback when no OIDC is configured but sandbox JWT signing IS — produces a synthetic dev-anonymous user principal so the rest of the handler chain treats CLI calls as User and bypasses the IDOR guard. Production OIDC deployments are unaffected: when OIDC is configured the fallback is not installed and missing-Bearer still 401s. 2. Sandbox supervisor re-ran the K8s SA bootstrap exchange on every connect_channel() call. With multiple subsystems each building their own channels, IssueSandboxToken was firing every few seconds even though TOKEN_SLOT already had a fresh token. Change connect_channel to reuse TOKEN_SLOT when populated; only run acquire_sandbox_token on the first call per process. The refresh loop keeps the slot fresh thereafter. 3. K8s SA authenticator looked up sandbox pods in the gateway's own namespace (POD_NAMESPACE) instead of the K8s driver's configured sandbox namespace. Source from kubernetes_config_from_file() so the resolver targets the same namespace the driver creates pods in. Verified end-to-end against the helm-dev cluster: - Two sandboxes get distinct gateway JWTs with their own sandbox UUIDs. - Cross-sandbox GetSandboxConfig is rejected with PermissionDenied and the auth::guard audit log fires with both principal and requested IDs. - RefreshSandboxToken mints a new JWT and revokes the old jti; the old token is then rejected with Unauthenticated: revoked token.
…testing
Adds a small subcommand to the supervisor binary that issues one-shot
sandbox-class RPCs against the gateway using the supervisor's existing
token-acquisition pipeline. Designed to be invoked via docker exec or
kubectl exec into a running sandbox to verify the per-sandbox identity
flow end-to-end without writing a custom test binary inside the sandbox
image.
Subcommands:
- get-sandbox-config --sandbox-id <UUID> — call GetSandboxConfig
- refresh — call RefreshSandboxToken
- show-token — print raw gateway JWT bytes
- show-principal — pretty-print decoded JWT claims
Verification flow this enables (Docker path):
docker exec sandbox-a openshell-sandbox debug-rpc show-principal
docker exec sandbox-a openshell-sandbox debug-rpc \
get-sandbox-config --sandbox-id <sandbox-b-uuid>
# → exit code 7 + "PermissionDenied: cross-sandbox access denied"
K8s path: same RPCs, kubectl exec instead.
show-token and show-principal intentionally don't trigger the K8s SA
bootstrap exchange — they only read an already-cached token, so
inspection doesn't burn a fresh JWT mint per call.
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
🌿 Preview your docs: https://nvidia-preview-pr-1414.docs.buildwithfern.com/openshell |
TaylorMutch
force-pushed
the
tmutch/per-supervisor-authn
branch
3 times, most recently
from
719f5f5 to
0d7df90
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add optional SPIRE/SPIFFE support for Helm dev clusters and allow sandbox supervisors to authenticate to the gateway with SPIFFE JWT-SVIDs instead of gateway-minted JWTs.
Related Issue
Stacked on #1404.
Changes
rust-spiffecrate for JWT-SVID fetch and validation, aligned with tonic/prost 0.14.Testing
RUSTC_WRAPPER= mise run pre-commitpassescargo check -p openshell-core -p openshell-server -p openshell-sandbox -p openshell-driver-kubernetespasses/helm-dev-environmentSPIRE deployment exercised sandbox list/create/delete and supervisor connect-backChecklist