capacity: grid-overflow residency stack — a persona's brain routes off-box (rescued from #2056) - #2250
Open
joelteply wants to merge 8 commits into
Open
capacity: grid-overflow residency stack — a persona's brain routes off-box (rescued from #2056)#2250joelteply wants to merge 8 commits into
joelteply wants to merge 8 commits into
Conversation
… capacity LeaseRequest (grid-overflow bridge)
The clean half of the grid-overflow seam. serving_plan reasons about MODEL
RESIDENCY (can a peer hold model M's weights + per-lane KV at the served
window, and how many lanes). capacity/ reasons about CONCURRENCY SPIKES
(does a peer have a free lane RIGHT NOW — LeaseRequest{want_concurrency,
spike_bytes}). They're orthogonal and compose — residency is the eligibility
gate, concurrency is the right-now admission. Neither absorbs the other.
grid_lease_request(served_window, demand_lanes) is the one-directional map
from the serving side into the capacity side: demand_lanes → want_concurrency
(floored at 1), and the prefill compute spike at the live served window →
spike_bytes (prefill_compute_reserve(window, 1) — the transient the peer must
have free to accept the hop, distinct from the resident weights+KV the
residency gate already proved). No transport, no placement policy here — just
the honest projection so the grid-overflow router can ask a residency-eligible
peer for a concurrency lease.
Test grid_lease_request_maps_demand_and_the_prefill_spike pins both mappings.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…or grid overflow (governor consumer slice 1)
Grid-overflow routing (spill a persona's generation to a peer) has TWO orthogonal
gates that COMPOSE, never absorb each other (settled with BigMama 2026-07-27):
(1) RESIDENCY = eligibility — does the peer already hold model M resident? If not,
accepting the hop forces a cold full-weights load (seconds to minutes), which
defeats overflowing for speed. So the fast path is eligible only for peers that
already hold M. THIS module.
(2) CONCURRENCY = right-now admission — does the peer have a free lane? capacity/grid
(LeaseRequest, LocalFirstFitPolicy). Unchanged.
The one crossing point between the two abstractions is ModelFootprint::grid_lease_request
(serving demand -> LeaseRequest) — one bridge, not two half-bridges. Residency deliberately
does NOT live on PeerCapacity (that would blur the concurrency abstraction with a residency
fact); it lives here as ModelResidencyView, keyed on Uuid exactly like gossip's capacity
ledger. The governor COMPOSES the two at the placement filter: residency_eligible() returns
a SMALLER snapshot (local untouched — the overflowing node holds M by definition; peers
filtered to those holding M), and the unchanged capacity policy places on the survivors and
applies reachability itself. Two concerns, composed at exactly one point, neither absorbed.
Latest-wins replace (not merge) so a model paged OUT stops being eligible on the next beacon
— a merge would resurrect evicted models and route a hop to a peer that no longer holds it.
3 tests: eligibility filter, latest-wins replace, and the residency->capacity compose
end-to-end (resident+reachable gets lanes; resident+unreachable reclaimed by place();
non-resident absent from placement). Zero blast radius — new file, no existing struct touched.
Next slice: populate the view from a residency beacon (gossip wiring, coordinated with
BigMama — piggyback CapacityOffer vs a separate slower-cadence stream).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…ct half of the residency beacon (governor consumer slice 2)
The residency sibling of capacity/gossip's CapacityOffer/GridCapacityLedger, with the
same identity + freshness discipline, a different (slower) cadence, and its own payload:
- ResidencyBeacon (wire): the model ids a node holds resident + a sender timestamp.
Rides its OWN grid_residency EphemeralCoalesced envelope — residency changes on model
page-in/out (minute-scale), NOT the 10s capacity beat, so coupling them would either
over-publish residency or under-refresh capacity. Peer identity is the WIRE's, never the
payload's — a peer cannot beacon residency on another's behalf.
- ResidencyLedger + global_residency_ledger(): folds heard beacons (latest-per-peer wins),
projects a ModelResidencyView, evicts beacons silent past RESIDENCY_EVICTION_WINDOW_MS,
excludes the node's own echo (local residency is its own serving truth). view() is the
exact residency analogue of GridCapacityLedger::snapshot().
Eviction is GENEROUS (6× the capacity window) precisely because the two abstractions stay
orthogonal: residency is sticky, and the COMPOSED capacity snapshot already gates reachability
— so a residency reading never has to prove liveness itself (that would blur residency into
concurrency). A long-silent peer falls back to UNKNOWN residency (not asserted-resident),
keeping the fast overflow path honest.
3 more tests (6 total in the module): heard-beacon-projects + own-echo-excluded (loopback),
stale-evict-then-fresh-restore, serde round-trip (camelCase wire, catches field drift).
Next slice: the publish + inbound-fold wiring — a GridResidencyModule mirroring
GridCapacityModule (build the beacon from the serving plan's resident set, broadcast
grid_residency) + the inbound_attach fold into global_residency_ledger(). Then a node
advertises its residency and residency_eligible() runs on live grid data.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…ency beacon publish/fold wiring (governor consumer slice 3)
Closes the residency-beacon loop end-to-end, mirroring the capacity gossip path
(GridCapacityModule + inbound_attach fold) exactly:
- GridResidencyModule (modules/grid_residency.rs): a Background ServiceModule that
every RESIDENCY_PUBLISH_INTERVAL_MS reads the daemon's live serving plan (lock-free
watch snapshot — the SAME source, no parallel probe) and broadcasts a ResidencyBeacon
over airc as an EphemeralCoalesced grid_residency envelope. Today the resident set is
[base_model_id]; a multi-model plan extends only current_beacon(). Honest silence when
no plan is computed yet (nothing to advertise). Glass box speaks on change.
- AircRealtimeSchema::GridResidency (airc/realtime.rs): the new schema variant,
EphemeralCoalesced like GridCapacity. ts-rs binding regenerated (AircRealtimeSchema.ts).
- inbound_attach fold: residency_beacon_from_envelope decoder + the else-if that folds a
heard beacon into global_residency_ledger(), keyed on the WIRE's peer id — the orthogonal
sibling of the capacity fold. Own echo lands here too (the single-node loopback proof).
- Registered in ipc/mod.rs right after GridCapacityModule, fed serving_daemon.subscribe().
- RESIDENCY_PUBLISH_INTERVAL_MS (model_residency.rs) = eviction/12 — the same
publish:eviction ratio capacity uses, on a slower beat (residency changes minute-scale).
This completes the grid-overflow governor-consumer stack: a node now ADVERTISES which
models it holds, every peer folds those beacons into a ModelResidencyView, and the governor
composes it with the capacity snapshot (residency_eligible -> grid_lease_request -> place ->
aircPeer hop). The live two-node routing smoke (BigMama's node up + serving) validates the
cross-node generation — the milestone this stack was built for.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…decision (governor consumer slice 4a)
The DECISION half of the driver, pure + fully unit-tested; only the thin EFFECT half
(the actual Commands.execute("ai/generate", {aircPeer}) hop) needs a live peer and lands
at the two-node smoke.
Overflow lanes are BY DEFINITION the ones ServingPlan.grid_overflow_lanes said couldn't fit
locally, so their placement is REMOTE-ONLY — it must never touch LocalFirstFitPolicy's
local-first >=1 floor (that floor is the local persona's OWN guaranteed lane, orthogonal to
spillover; re-cramming there is the exact thrash the honest overflow signal exists to avoid).
Confirmed with BigMama 2026-07-27.
Two orthogonal gates, composed (never absorbed):
1. RESIDENCY (ModelResidencyView::residency_eligible) — a peer is a fast overflow target
only for a model it ALREADY holds (else a cold full-weights load defeats the point).
2. CONCURRENCY (reachability + lanes_that_fit misfit-parts) — among reachable eligible
peers, most-free-first, each capped by its OWN budget for the prefill spike.
Unplaced lanes (no eligible+reachable peer could take them) are SURFACED in
OverflowRouting.unplaced for the caller to queue/degrade on — never silently dropped
([[fallbacks-are-illegal-fail-loud]]).
4 tests: remote-only + residency/reachability gating, unplaced-surfaced-not-dropped,
zero-overflow no-op, most-free-first spill spread.
This completes the pure governor-consumer decision path. Remaining (slice 4b, at the live
two-node smoke): read grid_overflow_lanes from the live plan, build the lease via
footprint.grid_lease_request, call route_grid_overflow, and execute the aircPeer hop per
placed (peer, lanes) — the only piece needing a real peer to route to.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…rflow_adapter_for override (governor consumer slice 4b-i) The composition point where route_grid_overflow's decision becomes a real remote brain. materialize_adapters gains an `overflow_adapter_for(&profile, slot)` closure (same closure-DI shape as runtime_lookup / tool_executor_for): return Some(remote adapter) when the governor routed this persona off-box — her node is over capacity and a reachable peer holds her model — so her brain runs on that peer via AircRemoteInferenceAdapter; None → build the local adapter from the factory (the common case). This is the exact re-home seam the DeliberationModelBinding was designed for (its doc: re-home = "a new adapter / grid failover onto another node"). The remote adapter registers in the global provider registry by model_id just like the local one, so evaluate_response reaches it transparently — the persona doesn't know or care that her inference crosses the grid. host.rs passes `|_,_| None` for now (slice 4b-ii wires the live capacity + residency + airc closure at the ipc bootstrap, where the serving plan + airc handle live). Test overflow_effector_supplies_remote_adapter_and_bypasses_the_local_factory pins the contract: when the override supplies a slot's adapter, the local factory is NOT called for it (build_count == 1 of 2), both personas host, both warm. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…n routes off-box to a residency-eligible peer (governor consumer slice 4b-ii) The last mile. build_overflow_effector (persona/grid_overflow_effector.rs) composes the whole tested decision path into the per-persona adapter override the supervisor consumes, wired at the ipc bootstrap spawn: live serving plan (grid_overflow_lanes) → footprint from live_candidates → grid_lease_request → global_residency_ledger().view → route_grid_overflow → AircLiveTransport(airc, peer) → AircRemoteInferenceAdapter When the node is over local capacity and a reachable peer already holds a persona's model, her DeliberationModelBinding.adapter becomes the airc-remote one — her inference crosses the grid transparently (the re-home the binding was designed for), and she lives in the room as a peer hosted on another machine. That's "a competent peer so it's not just us there." DEFENSIVE by construction (safe to ship pre-smoke): returns None → local adapter on ANY uncertainty (airc not attached, no plan, no overflow, no matching footprint, no eligible reachable peer). Can only be a safe no-op or a correct off-box route — never a self-route (own peer excluded via airc.peer_id(), so its own residency-beacon loopback can't pick itself) and never a panic. The only unit-unprovable part is that the remote hop SUCCEEDS — the live two-node smoke validates that; a hop that can't warm surfaces as a loud AdapterWarmup slot failure, never a silent local downgrade ([[fallbacks-are-illegal-fail-loud]]). Plumbing: overflow_adapter_for threaded through spawn_all → materialize_adapters (4b-i seam); dedicated Arc clones of the airc-interceptor cell + serving daemon so the boot-spawn async-move capture doesn't strand the interceptor + reconcile task; live_candidates() → pub(crate). Completes the grid-overflow governor consumer end-to-end. Live validation + the cross-node generation run the moment BigMama's node is serving (model_id string-equality is the one thing to confirm live). All unit paths green (21 supervisor/overflow/residency tests). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…t over it Rebase-resolution fix, and a correction of my own error worth recording. While cherry-picking the residency stack I hit a conflict in serving_plan.rs and resolved it with `git checkout --theirs` — which takes the WHOLE branch file. That silently discarded canary's ServingDemand (the elastic-window struct that landed separately), producing 'unresolved import ServingDemand' in two modules. Choosing a side is not merging; it is overwriting with extra steps, and it is the same class of mistake as blind-skipping commits during a rebase. Correct resolution: keep canary's serving_plan.rs verbatim and add ONLY the one symbol the residency stack actually needs — ModelFootprint::grid_lease_request, the serving→grid bridge that maps a footprint to a capacity LeaseRequest (want_concurrency = demanded lanes, spike_bytes = ONE lane's prefill reserve at the served window). cargo check -p continuum-core --lib --tests clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc
Contributor
Author
|
Rebase attempt 2026-08-14: 24 conflict hunks in cognition/serving_plan.rs, and they are SEMANTIC — this branch predates the ServingDemand refactor and encodes the reverted MAX_LANES=2 expectations, while canary now derives warm slots per resident persona (#213/#234/#266 line). A mechanical resolution would silently pick one serving-economics story over the other. Needs a deliberate rework against current canary: keep the PR's unique addition (ModelFootprint::grid_lease_request bridge) and rewrite its tests against the ServingDemand API. Deferring rather than guessing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fourth rescue from #2056. Seven commits, cherry-picked onto canary, verified.
What lands
The grid-overflow residency path, end to end:
ModelFootprint::grid_lease_request— serving→grid bridge (footprint → capacityLeaseRequest)ModelResidencyView— the residency eligibility gateResidencyBeacon+ResidencyLedger— the receive/project halfGridResidencyModule+grid_residencyenvelope — the beacon publisher, reading the SAME serving-plan watch snapshot the daemon computes (no parallel probe)route_grid_overflow— remote-only overflow placement decisiongrid_overflow_effector.rs, a persona's brain routing off-boxComplements #2246 (division actuation): that wires the governor's VRAM division locally; this is the spill path when local won't hold it.
A resolution error I made and corrected in-branch
The serving_plan.rs conflict I first resolved with
git checkout --theirs, which takes the whole branch file and silently discarded canary'sServingDemand— surfacing asunresolved import ServingDemandin two modules. Choosing a side is not merging. It is the same class as blindrebase --skip: fast, plausible, and it deletes work.Correct resolution is the final commit: canary's
serving_plan.rskept verbatim, with onlygrid_lease_requestmerged in — the one symbol this stack actually needs.Verification
cargo check -p continuum-core --lib --testsclean.