vMCP: backend health/set changes are not propagated to live sessions (stale tool catalog until reconnect; no tools/list_changed)

[premctl]

vMCP: backend health/set changes are not propagated to live sessions (stale tool catalog until client reconnect; no tools/list_changed)

Type: bug / enhancement · Area: vMCP (aggregator / server / health) · Version: v0.34.0 (latest release; also on main)
Related: companion to the SSRF-allowlist request (co-located backends) — same deployment.

Summary

When a vMCP backend's availability changes at runtime — a backend recovers (unhealthy→healthy), or is added/removed from the group — already-connected client sessions do not see the change. Their tool catalog stays as it was when the session was created, and the vMCP never emits notifications/tools/list_changed. Clients must reconnect (re-initialize the session) to pick up a newly-healthy backend's tools. New sessions are fine; existing ones are stuck.

Repro

1. Run a vMCP fronting ≥2 backends; connect a client (session A).
2. A backend that was unhealthy at A's connect time becomes healthy (fix its endpoint, restart it, etc.). The vMCP re-aggregates — logs capability aggregation complete backends:N tools:M with the new backend included.
3. From session A: the new backend's tools are not callable (call_tool → tool not found) and don't appear in discovery. A freshly connected session B sees them.
4. Only reconnecting session A surfaces them.

(In optimizer mode this is extra-confusing: the client's advertised tools are the stable find_tool/call_tool meta-tools, so even a client that honors tools/list_changed has nothing new to re-fetch — the staleness lives inside the optimizer catalog that find_tool/call_tool resolve against.)

Root cause (verified, v0.34.0)

• Per-session snapshot, taken once at registration. pkg/vmcp/server/serve_handlers.go → injectCoreSessionCapabilities → coreSessionTools calls s.core.ListTools(...) only when the session is registered, and installs the result via setSessionToolsDirect. In optimizer mode serveSessionTools builds the find_tool/call_tool meta-tools over that same one-time snapshot.
• Aggregation has no downstream notification. pkg/vmcp/aggregator/default_aggregator.go (AggregateCapabilities) and pkg/vmcp/aggregator/caching_aggregator.go compute/refresh the merged set, but neither notifies sessions nor the server of a change.
• Health monitor detects transitions but doesn't fan out. pkg/vmcp/health/monitor.go tracks status (statusTracker, UpdateBackends, StatusProvider/Reporter interfaces) and logs starting health monitoring for new backend, but there is no callback/subscription that re-injects live sessions or triggers a client notification on a state change.
• No tools/list_changed is ever emitted on the vMCP path. The method is parsed (pkg/mcp/parser.go) and the primitive exists (pkg/transport/bridge.go → SendNotificationToAllClients), but it is only used by the single-server proxy bridge, not wired into the vMCP aggregation/session layer.

Net: health monitor → re-aggregate is a dead end w.r.t. connected sessions.

Proposal

On a backend-set change (health transition, or group add/remove), propagate to live sessions:

1. Passthrough mode (optimizer off): update each connected session's advertised tools (the go-sdk supports per-session AddSessionTools/SetSessionTools/DeleteSessionTools) and emit notifications/tools/list_changed to all clients via the MCPServer (the same primitive the bridge already uses). Requires the server to advertise tools.listChanged: true.
2. Optimizer mode (optimizer on): rebuild the optimizer catalog/index that find_tool/call_tool resolve against so existing sessions resolve the new backend without reconnect (either refresh the per-session optimizer state, or have the optimizer resolve against the live core catalog on each call / on a change signal).
3. Wiring: give health.Monitor a change-callback / subscription (it already centralizes status via statusTracker); the core/server subscribes, re-aggregates, and performs (1)/(2). Debounce to the existing statusReportingInterval so a flapping backend doesn't storm sessions.

Notes

• Gate strictly on the server advertising tools.listChanged; no behavior change otherwise.
• The debounce already present for health (healthCheckInterval × unhealthyThreshold) keeps this from firing on transient blips.
• Happy to help / open a PR — a sketch of the integration points is written up alongside this.

[premctl]

PR sketch — propagate vMCP backend-set changes to live sessions

Addresses #5786 (companion to #5784). Goal: when the healthy backend set changes at runtime (unhealthy→healthy, or group add/remove), connected sessions see the new/removed tools without reconnecting, and the vMCP emits notifications/tools/list_changed.

Current flow (verified, v0.34.0)

health.Monitor (statusTracker)  ──detects transition──▶  (logs only, dead end)
core.AggregateCapabilities  ──build catalog+routing──▶  cachingAggregator (cached)
session register ──▶ injectCoreSessionCapabilities ──▶ coreSessionTools ──▶ core.ListTools
                     └─ optimizer on: build find_tool/call_tool over that ONE-TIME snapshot

SendNotificationToAllClients exists (pkg/transport/bridge.go) and tools/list_changed is parsed (pkg/mcp/parser.go), but neither is wired to the aggregation/session path.

Proposed flow

health.Monitor ──OnChange(gen)──▶ core (re-aggregate, debounced) ──catalog(gen)──▶ server
     server:  for each live session → re-apply tools (setSessionToolsDirect / optimizer rebuild)
              → mcpServer.SendNotificationToAllClients("notifications/tools/list_changed", nil)

Changes by file

1. pkg/vmcp/health/monitor.go — surface transitions

• Add a change hook to Monitor (it already centralizes state in statusTracker and handles group edits in UpdateBackends):

  type ChangeListener func(gen uint64)          // gen = monotonically increasing generation
  func (m *Monitor) OnChange(fn ChangeListener)  // register

• Fire it from statusTracker on a healthy⇄unhealthy transition and from UpdateBackends on add/remove. Coalesce/debounce to statusReportingInterval so a flapping backend doesn't storm.

2. pkg/vmcp/core/core_vmcp.go — re-aggregate on signal

• On OnChange, recompute aggregator.AggregateCapabilities(ctx, healthyBackends) and swap the core's catalog + routing table atomically; bump a generation. (The cachingAggregator cache must be keyed on / invalidated by the healthy-set generation so it doesn't serve the stale merge.)
• Expose the change to the server via a callback or a <-chan generation.

3. pkg/vmcp/server/serve_handlers.go (+ session registry) — apply to live sessions

• Split the existing per-session build so it can run post-init: factor the body of injectCoreSessionCapabilities/coreSessionTools into an applySessionCapabilities(session) that diffs current vs new.
• On a core generation bump, for each tracked live session:
  • passthrough: recompute coreSessionTools and re-apply via the existing setSessionToolsDirect (add new, remove gone).
  • optimizer: rebuild the find_tool/call_tool meta-catalog for that session from the new core.ListTools (this is the half that actually fixes optimizer mode — advertised tools stay {find_tool, call_tool}, but their backing index refreshes).
• After applying, s.mcpServer.SendNotificationToAllClients("notifications/tools/list_changed", nil).
• Advertise tools.listChanged: true in server capabilities (gate: only notify if advertised).

Concurrency / correctness

• Single writer: the core owns the catalog + generation; sessions are updated under their existing session lock. Skip a session whose applied generation already == current.
• Routing table swap must be atomic with the catalog so an in-flight call_tool never resolves a name against a mismatched route.

Backward compatibility

• No-op unless the server advertises tools.listChanged. Optimizer-catalog refresh is internal (no protocol change). Existing debounce keeps blips from propagating.

Tests

• unit: monitor fires OnChange exactly once per debounced transition; core re-aggregates and bumps generation; cachingAggregator invalidates on new generation.
• server unit: on generation bump, a live session gains/loses the right tools; SendNotificationToAllClients called once.
• e2e (test/e2e/thv-operator/virtualmcp/): connect session A; flip a stub backend unhealthy→healthy; assert (a) A receives tools/list_changed, (b) the new backend's tool is callable on A without reconnect — run the matrix in both passthrough and optimizer modes.

Scope / risk

• Focused change across monitor→core→server; no CRD/API change. Main risk is the atomic catalog/routing swap under concurrent calls — covered by the routing-consistency test. Ship behind the listChanged capability so it's opt-in by server config.

[aponcedeleonch]

Thanks for the incredibly thorough writeup. I walked all four paths on current main (75b3ed7) and the root-cause analysis holds up:

• injectCoreSessionCapabilities -> coreSessionTools -> core.ListTools runs once per session from handleSessionRegistrationImpl, and the optimizer find_tool/call_tool meta-tools in serveSessionTools get built over that same one-time snapshot. setSessionToolsDirect even bypasses AddSessionTools on purpose to avoid notifications.
• The aggregators (default_aggregator.AggregateCapabilities, caching_aggregator) only log and cache. No downstream notify.
• Health transitions in statusTracker.RecordSuccess/RecordFailure are log-only, and UpdateBackends is only called from the periodic reportStatus, which never re-injects sessions.
• SendNotificationToAllClients is wired only into the single-server proxy bridge (pkg/transport/bridge.go), not the vMCP path.

So health monitor -> re-aggregate really is a dead end for connected sessions. I didn't stand up a live multi-backend vMCP to reproduce the flap, since the gap is structural and the code paths settle it, but flagging that the repro here is by code inspection, not a live run.

Confirming this as a valid open issue. Your integration sketch lines up with how the code is actually structured.

One suggestion on implementation: I'd split this into two PRs rather than one, since the two modes have pretty different blast radius.

• PR 1, passthrough mode (optimizer off): wire the health monitor change-callback -> core re-aggregate -> per-session re-apply, update each connected session's advertised tools, and emit notifications/tools/list_changed, gated on the server advertising tools.listChanged. This is the more contained change and a good place to start.
• PR 2, optimizer mode: rebuild the optimizer catalog/index that find_tool/call_tool resolve against so existing sessions pick up the new backend without reconnect. This one has more implications, so it's worth isolating. The catalog/routing swap needs to be atomic so an in-flight call_tool never resolves against a mismatched route, plus there's the per-session optimizer state to think through.

Happy to review both if you're up for taking them up.

[premctl]

Thanks for walking all four paths — and good catch that setSessionToolsDirect bypasses AddSessionTools specifically to suppress notifications; that's the exact seam PR1 has to change.

The two-PR split makes sense — the blast radius is very different:

• PR1 (passthrough): health.Monitor change-callback → core re-aggregate → per-session re-apply advertised tools + SendNotificationToAllClients("notifications/tools/list_changed", …), gated on the server advertising tools.listChanged. Contained and independently useful.
• PR2 (optimizer): rebuild the find_tool/call_tool catalog for live sessions, with the atomic catalog/routing swap so an in-flight call_tool never resolves against a mismatched route.

One design question for PR1: do you want the change signal modeled as a callback registered on Monitor, or a <-chan the core selects on (fits the existing reportStatus loop)? I'll match whatever suits the codebase.

Assuming we take these up, I'd start with PR1.