feat: detect and resolve middleware manifest drift with digest pinning and gateway refresh

[pimlock]

Problem Statement

The gateway builds its middleware registry once at startup and caches every service manifest for its whole lifetime. The sandbox supervisor, by contrast, re-fetches manifests via Describe at sandbox startup and on every registry rebuild. When an operator upgrades a middleware service without restarting the gateway, the two views drift, and each direction of drift has a bad failure mode:

• Added binding: the gateway rejects policies referencing it ("not registered") even though the live service supports it. The only fix is a gateway restart.
• Removed or renamed binding: the gateway may still admit policies referencing the stale id (its cached manifest contains it; only a well-behaved service's live ValidateConfig catches it). On the supervisor's next registry rebuild the fresh manifest lacks the binding, the chain entry becomes unresolved, and a fail_open guard is bypassed per request. This is only observable after unguarded traffic has already flowed, via per-request Medium DetectionFindings that are indistinguishable from a transient service error, so nothing attributes the bypass to manifest drift.
• Changed max_body_bytes or config semantics: the supervisor keeps using the manifest captured at its last rebuild; a service that shrank its limit errors on oversized bodies and degrades per on_error.

There is currently no install-time detection of any of this and no way to refresh the gateway's view short of a restart.

Proposed Design

Three layers, each independently useful, ordered by cost:

1. Supervisor-side binding cross-check at registry rebuild (no protocol change)

Inside the combined policy+registry reload, after connect_services returns the freshly described registry, call ensure_policy_bindings_registered against the new registry for the incoming policy. A snapshot whose policy references bindings the live manifests no longer expose is rejected like any other reload failure: the last-known-good runtime pair stays installed, the snapshot is retried each poll, and the failure is loud (Medium OCSF config-state event plus failed policy status reported to the gateway). This converts the silent fail_open bypass at install time into a blocking, attributable signal.

2. Manifest digest pinning in the delivered registration (small, backward-compatible protocol addition)

Extend SupervisorMiddlewareService (sandbox.proto) with a gateway-stamped manifest_digest and the manifest's service_version:

• The digest is computed over the canonicalized manifest: bindings sorted by id, hashing id, operation, phase, and max_body_bytes. service_version is carried for diagnostics only (it is a free-form string and must not be the comparison key).
• During a registry rebuild, the supervisor compares the live manifest digest from its fresh Describe against the delivered digest. On mismatch it fails the snapshot (keep last-known-good, retry) and emits a DetectionFinding naming the registration, both digests, and both service versions, so operators see "the middleware contract changed underneath the gateway" rather than a generic connect error.
• Backward compatibility: an empty digest skips the check, so old gateways keep working with new supervisors and vice versa.

This also buys fleet convergence for free: the digest rides in the delivered registration, which is hashed into config_revision, so whenever the gateway's view of a manifest changes, every sandbox observes a service-set change on its next poll and performs a combined rebuild with a fresh Describe.

3. Gateway periodic manifest refresh

Re-Describe registered services on an interval, retaining the last-known-good manifest when a Describe fails so a flapping service cannot invalidate admitted policies. Combined with layer 2, a middleware upgrade propagates automatically: gateway refresh notices the new manifest, the delivered digest changes, config_revision changes, and every affected sandbox rebuilds against the new contract or loudly rejects it where it breaks the policy. This also removes the gateway-restart requirement for added bindings. Note one intentional behavior change: after a refresh removes a binding that admitted policies still reference, the per-delivery ensure_policy_bindings_registered check starts failing config delivery for those sandboxes (fail-safe, but it blocks their settings updates until the policy or service is fixed, and it should raise a gateway-side alert).

Alternatives Considered

• Pin service_version instead of a manifest digest: rejected as the comparison key. It is a free-form diagnostics string, so it over-detects (patch release with identical bindings) and under-detects (bindings changed without a version bump). Kept in the finding message for humans.
• Supervisor heartbeat re-Describe to catch mid-life drift: rejected. The per-request fail-open DetectionFinding already provides mid-life visibility, and layer 3 handles convergence at the right layer without a second polling loop per sandbox.
• Rely on the existing per-request fail-open findings alone (status quo): they fire only after unguarded requests have flowed, scale with traffic instead of with the drift event, and cannot be attributed to drift.
• Document "restart the gateway after middleware upgrades" (status quo): works but is easy to forget, and the failure mode for forgetting is a silently bypassed guard.

Agent Investigation

• Gateway registry is built once at startup and never refreshed: crates/openshell-server/src/lib.rs (hard failure if a registered service is unreachable at startup). Manifests are cached in a OnceCell per service, populated eagerly by connect_services.
• Policy admission validates against the cached manifest and calls the live ValidateConfig: crates/openshell-server/src/middleware.rs, MiddlewareRegistry::validate_policy_configs.
• Every config delivery re-checks ensure_policy_bindings_registered and filters required_services, both against the startup manifest snapshot: crates/openshell-server/src/grpc/policy.rs (get_sandbox_config).
• config_revision already hashes the delivered supervisor_middleware_services, which is what makes layer 2's convergence property work with zero new plumbing: compute_config_revision in crates/openshell-server/src/grpc/policy.rs.
• The supervisor rebuilds its registry with fresh Describe calls only when the delivered service set changes or the installed registry is degraded; policy-only changes swap the OPA engine alone (see PR feat(supervisor-middleware): add network egress middleware #2027).
• The delivered registration currently carries only name, grpc_endpoint, and max_body_bytes: proto/sandbox.proto (SupervisorMiddlewareService).
• Per-request fail-open bypass finding that provides today's only drift visibility: crates/openshell-supervisor-network/src/l7/middleware.rs (middleware_events, "Middleware failed and was bypassed (fail_open)").