From 9b8b8468e744fc4a0e48f03d8e92535152f1ae7a Mon Sep 17 00:00:00 2001 From: Joel Teply Date: Sat, 18 Jul 2026 16:39:29 -0500 Subject: [PATCH 1/5] =?UTF-8?q?fix(airc-purity):=20route=20voice-mirror=20?= =?UTF-8?q?warn=20via=20clog=5Fwarn=20=E2=86=92=20live.log;=20LIVE-VALIDAT?= =?UTF-8?q?ED=20(#193)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The #193 divergence warn now uses clog_warn! (auto-routes by module_path to live.log — the SAME sink the orchestrator's register logs use), replacing log_warn!/tracing::warn! which routed to sinks I never found. Also the FIRST end-to-end proof of the #194 deploy guard: this new behavior actually SHIPPED through the guarded start-server.sh and ran. LIVE RECEIPT (running core, this branch): DIVERGENT session_id != room_id → [WARN] live: MIRROR ROOM (#193): ... dddddddd != eeeeeeee — the LiveKit call is keyed by a client-minted session_id, not the room_id. COLLAPSED session_id == room_id → SILENT (0 MIRROR lines). So the instrument works: fires on the parallel-room-authority, silent when the call IS the airc room. It is the glass-box we watch go quiet when the actual collapse lands. Refs #193, #194. [[all-rooms-are-airc-rooms-no-mirrors]], [[verify-the-build-actually-deployed]] Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo --- core/continuum-core/src/modules/live.rs | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/core/continuum-core/src/modules/live.rs b/core/continuum-core/src/modules/live.rs index 18bfbd574b..825287302a 100644 --- a/core/continuum-core/src/modules/live.rs +++ b/core/continuum-core/src/modules/live.rs @@ -108,9 +108,11 @@ impl ServiceModule for VoiceModule { // behaviour change; a pure diagnostic until the client-coordinated cutover lands // ([[all-rooms-are-airc-rooms-no-mirrors]], [[livekit-media-plane-rides-airc-not-parallel]]). if session_id != room_id { - crate::log_warn!( - "module", - "voice_register_session", + // clog_warn! auto-routes by module_path to the module's log file (live.log) — + // the SAME sink the orchestrator's register logs land in — so the divergence is + // observable next to the registration it describes. (log_warn!/tracing::warn! + // routed elsewhere and were invisible; #194 also masked this by shipping stale.) + crate::clog_warn!( "MIRROR ROOM (#193): voice session/call id {} != airc room id {} — the LiveKit \ call is keyed by a client-minted session_id, not the airc room_id. Collapse: \ the client must pass session_id == room_id so the call IS the airc room.", From fc84a3e02e11a25291825a4a81b2f14fdd36c56d Mon Sep 17 00:00:00 2001 From: Joel Teply Date: Sat, 18 Jul 2026 16:49:46 -0500 Subject: [PATCH 2/5] =?UTF-8?q?docs(plan):=20LiveKit=20perception=20pipeli?= =?UTF-8?q?ne=20=E2=80=94=20the=20critical=20path=20to=20personas=20seeing?= =?UTF-8?q?=20a=20live=20call?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The methodical plan to get personas perceptive through LiveKit: frame → MediaFrame (content-hash) → deep cells (describe/pose/detect, computed once, shared) → per-persona projection → cognition → perceive/speak. Decisions baked in: Qwen3-VL-7B local as first eyes; ambient + drill-in intake. 6 phases, each with a validation gate; critical path = Phase 1 (Qwen3-VL up) → Phase 2 (video_rx → MediaFrame) → Phase 3 (render wire #190, the 'persona describes the call' milestone). #193 (call = airc room) runs parallel for identity/purity. Grounded in what's already in tree — wiring + one model bring-up. Refs #106, #187, #190, #192, #193. [[media-is-compute-once-zero-copy-hardware-grade]] Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo --- docs/planning/LIVEKIT-PERCEPTION-PIPELINE.md | 111 +++++++++++++++++++ 1 file changed, 111 insertions(+) create mode 100644 docs/planning/LIVEKIT-PERCEPTION-PIPELINE.md diff --git a/docs/planning/LIVEKIT-PERCEPTION-PIPELINE.md b/docs/planning/LIVEKIT-PERCEPTION-PIPELINE.md new file mode 100644 index 0000000000..88974c2e03 --- /dev/null +++ b/docs/planning/LIVEKIT-PERCEPTION-PIPELINE.md @@ -0,0 +1,111 @@ +# LiveKit Perception Pipeline — personas that see, hear, and describe a live call + +**Status:** build plan (Joel + Claude, 2026-07-18). The methodical critical path to +"a persona in a LiveKit call perceives the other participants — appearance, gestures, +scene/what's-presented — on ANY base model." Companion to +[PERCEPTION-SURFACE.md](../architecture/PERCEPTION-SURFACE.md) (#187), +[the media substrate](../../core/continuum-core/src/media/) (compute-once frames), +and the airc-room call collapse (#193). + +## Goal + +A persona in a call **perceives**: sees each participant (identity/appearance, gestures +like a raised hand), describes the scene / what's being presented, hears the audio, and +speaks back — with **equal sensory access** across base models (native-vision reads the +pixels; non-vision reads the TEXT of the same cached cells). The 13-persona video wall is +the scaling target; two personas is the first proof. + +## The pipeline (one line) + +``` +LiveKit frame → MediaFrame (content-hash) → deep cells (describe / pose / detect, + computed ONCE, shared) → per-persona projection → cognition context → perceive + → speak (TTS) / act +``` +Audio runs the parallel loop: `LiveKit audio → STT → cognition → TTS → LiveKit`. + +## Decisions (2026-07-18) + +- **First eyes: Qwen3-VL-7B, served LOCAL** (GGUF + mmproj in llama-server). Fits a + MacBook, native vision, our exact stack; it is BOTH the describer for non-vision + personas AND the native-vision base. Bigger tiers (32B on the 5090) and cloud VLMs come + later; 7B is the first-light proof. +- **Intake shape: AMBIENT + DRILL-IN.** Ambient = the persona periodically "looks"; the + current frame's cells (scene/appearance/gesture) enter its context each cadence tick, so + awareness is continuous and lifelike. Drill-in = a tool (`perception/observe-frame` or + equivalent) to look CLOSER at a specific tile/gesture on demand. Ambient low-res always + on; the tool for detail. + +## Current state (what's already real — don't rebuild) + +- **Deploy trust (#194)** ✅ — `core ready` no longer lies (freshness guard). Precondition + for ALL live validation this plan depends on. +- **Media substrate** ✅ (merged #1954) — `MediaFrame` (content-hash), `project_image` + (capability-gated, resolution-knobbed), the description cell + `FrameDescriber` trait, + compute-once/share on `SharedCompute`. +- **VL serving hook** ✅ (merged #1955) — `llama-server --mmproj` wired + a serving-time + `resolve_mmproj_for_model`. A VL model WOULD see; none is running yet. +- **LiveKit media plane** — `live/transport/{call_server,bridge_client}.rs` already surface + per-participant `video_rx` / `audio_rx` / `transcription_rx`. Frames arrive; nothing turns + them into perception yet. +- **The call is NOT an airc room yet (#193)** — glass-box instrument live-validated + (fires on `session_id != room_id`). + +## Phases (critical path bolded) + validation gate per phase + +**Phase 1 — Qwen3-VL-7B actually serving (#106 bring-up). THE EYES.** +Pull `Qwen3-VL-7B-Instruct` GGUF + its `mmproj`, bring up in llama-server, register the +`Vision` capability + the mmproj path on the Model row. Bounded and validatable ALONE — +nothing perceives without a describer. +*Gate:* `cu ai/generate` (or `cognition/vision-describe`) with an image → a real +description returns from the local model. + +**Phase 2 — Frame ingest: LiveKit video → `MediaFrame`.** +Tap the existing per-participant `video_rx`; wrap sampled frames as content-hash +`MediaFrame`s on the runtime `SharedCompute`. THROTTLE to a perception cadence (sample, not +30fps). One content-hash per distinct frame ⇒ N viewers share one set of cells. +*Gate:* a `cu` probe shows call frames landing as content-hashed frames + cache key counts. + +**Phase 3 — Perception into cognition (the render wire, #190). THE KEYSTONE.** +Wire the description cell (Qwen3-VL over the frame) into the persona's cognition context, in +the AMBIENT + DRILL-IN shape: ambient = inject the current frame's projected percept +(scene/appearance/gesture text or thumbnail, per capability) as a perception fact each +cadence tick; drill-in = a `perception/observe-frame` tool for a closer look. This is the +#190 render-seam wire (needs executor + `SharedCompute` threaded into the cognition-respond +path — STOP-zone plumbing; do the required PERSONA-COGNITION-PIPELINE read first). +*Gate:* a persona in a call, asked "what do you see," describes the current frame — and a +non-vision persona describes the SAME scene from the cached text cell. + +**Phase 4 — Deep cells: gesture / appearance / detection.** +Pose cell (hand-up, pointing), identity link (this tile = Atlas, via the avatar/name +anchor + #193 room roster), object/scene detection — each an async `SharedCompute` +derivative, computed once per frame, bridged in as it resolves (never stalls the turn). +*Gate:* a persona reports "Atlas has his hand up" from a frame with a raised hand. + +**Phase 5 — Audio loop (STT/TTS through the call).** +STT in → cognition → TTS out, distinct gender-matched voices. Partly wired +(`VoiceOrchestrator`); binds to the airc-room call (#193). +*Gate:* two personas exchange spoken turns in a call, each with a distinct matched voice. + +**Phase 6 — The proof (#192).** +Two personas in a call describe what the other shows + distinct voices → scale N +(compression: one describe per frame shared to all) → presentations (screen-share +perception) + collaborative webdev. + +## Cross-cutting parallel track: #193 (call = airc room) + +Orthogonal to *perceiving* but required for *identifying each other* (who's who = the airc +room roster) and for purity ([[all-rooms-are-airc-rooms-no-mirrors]]). Proceeds in parallel; +perception doesn't block on it, but Phase 4's appearance-identity grounds on it. Glass-box +instrument already live (#1957). + +## Sequencing + +1. **Phase 1 (Qwen3-VL-7B up)** — the concrete next build; validatable in isolation. +2. **Phase 2 (frame ingest)** — connect the existing `video_rx` to `MediaFrame`. +3. **Phase 3 (render wire #190)** — the milestone: "a persona describes the call." +4. Phases 4/5/6 stack; #193 runs alongside. + +Everything composes from pieces already in tree — this is wiring + one model bring-up, not +new physics. Prove each gate before the next ([[verify-the-build-actually-deployed]], +[[never-blind-feedback-driven-iteration]]). From 967efac2b7b9787b97d6c742669424f9fb5ce78a Mon Sep 17 00:00:00 2001 From: Joel Teply Date: Sat, 18 Jul 2026 16:51:09 -0500 Subject: [PATCH 3/5] docs(plan): ambient look = ~480w thumbnail, drill-in for bigger, resolution as context config; KV/memory reuse the hard constraint Joel: the forced-look frames are thumbnails (~480w) so ambient perception is cheap context; personas request bigger via the drill-in tool, or it's a per-persona/situation context config. Must be smart given memory + KV reuse. Baked into the plan: one scaled cell per frame content-hash shared to all viewers, image-token KV encoded ONCE and reused across ticks + viewers ('thumbnails into KV once for all'); sampling cadence, not 30fps. The ambient look is nearly free on the 2nd..Nth consumer by design. Refs #190, #192. [[perception-feedback-must-not-blow-rag]], [[media-is-compute-once-zero-copy-hardware-grade]] Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo --- docs/planning/LIVEKIT-PERCEPTION-PIPELINE.md | 25 ++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/docs/planning/LIVEKIT-PERCEPTION-PIPELINE.md b/docs/planning/LIVEKIT-PERCEPTION-PIPELINE.md index 88974c2e03..d3b55dde4c 100644 --- a/docs/planning/LIVEKIT-PERCEPTION-PIPELINE.md +++ b/docs/planning/LIVEKIT-PERCEPTION-PIPELINE.md @@ -36,6 +36,31 @@ Audio runs the parallel loop: `LiveKit audio → STT → cognition → TTS → L equivalent) to look CLOSER at a specific tile/gesture on demand. Ambient low-res always on; the tool for detail. +## Resolution & context budget — the ambient look is CHEAP by design + +The "forced look" must not blow context, memory, or KV — perception is continuous, so it +has to be small by default and reused hard ([[perception-feedback-must-not-blow-rag]], +[[media-is-compute-once-zero-copy-hardware-grade]], [[media-context-is-graduation-gated-scaffold]]): + +- **Ambient default ≈ 480px-wide thumbnail.** The forced-look frame is a small standard + scaled cell (`project_image` → `MediaResolution::Scaled(DestSize)`, ~480w). Little + context per tick — a native-vision persona gets ~one small image; a non-vision persona + gets the cached description text. Never the full-res frame on the ambient path. +- **Request bigger = the drill-in tool.** When a persona needs detail (read a slide, check + a gesture), it calls the drill-in tool for a larger cell — up to its model+adapter max, + derive-not-clamp. Full res is reachable, never forced. +- **Resolution is a CONTEXT CONFIG, not a constant.** The ambient default (~480w) is a + per-persona / per-situation knob: a big-context native-vision persona in a design review + may run a larger ambient; a tiny local model runs description-only. Derived from the + persona's real model+adapter + role + budget, threaded by reference — no hardcoded clamp + ([[no-hardcoded-context-numbers-derive-from-the-live-window]]). +- **KV / memory reuse is the hard constraint.** The 480w thumbnail is ONE scaled cell per + frame content-hash, shared to every viewer (13 personas, one cell). Its image-token KV + is encoded ONCE and reused across cadence ticks AND across viewers — "thumbnails into KV + once for all." Sampling cadence (not 30fps) + one-cell-per-content-hash + shared KV is + what makes the N-persona wall affordable. The ambient look is designed to be nearly free + on the second-and-Nth consumer. + ## Current state (what's already real — don't rebuild) - **Deploy trust (#194)** ✅ — `core ready` no longer lies (freshness guard). Precondition From 28f6a04e825ecf97ad1e257e113203e47072c29d Mon Sep 17 00:00:00 2001 From: Joel Teply Date: Sat, 18 Jul 2026 16:55:47 -0500 Subject: [PATCH 4/5] =?UTF-8?q?docs(plan):=20Phase=201=20SELF-PROVISIONS?= =?UTF-8?q?=20the=20VL=20model=20=E2=80=94=20no=20manual=20pull=20(managed?= =?UTF-8?q?=20product)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Joel: a repo user isn't asking Claude to run huggingface-cli. Corrected Phase 1: the VL bring-up self-provisions via the EXISTING provisioning::Downloader / models/pull path (GGUF + mmproj from the Model row's hf_source, no CLI dependency); mmproj resolution must match the GGUF's local→HF-cache→pull (today resolve_mmproj_for_model only checks the declared local path — the gap that forces manual placement). Gate: clean machine + declared VL row → auto-provision → vision-describe works, ZERO manual steps. Refs #106. [[managed-product-everything-self-provisions-no-operator-steps]] Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo --- docs/planning/LIVEKIT-PERCEPTION-PIPELINE.md | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/docs/planning/LIVEKIT-PERCEPTION-PIPELINE.md b/docs/planning/LIVEKIT-PERCEPTION-PIPELINE.md index d3b55dde4c..dff2ec7d5b 100644 --- a/docs/planning/LIVEKIT-PERCEPTION-PIPELINE.md +++ b/docs/planning/LIVEKIT-PERCEPTION-PIPELINE.md @@ -79,11 +79,19 @@ has to be small by default and reused hard ([[perception-feedback-must-not-blow- ## Phases (critical path bolded) + validation gate per phase **Phase 1 — Qwen3-VL-7B actually serving (#106 bring-up). THE EYES.** -Pull `Qwen3-VL-7B-Instruct` GGUF + its `mmproj`, bring up in llama-server, register the -`Vision` capability + the mmproj path on the Model row. Bounded and validatable ALONE — -nothing perceives without a describer. -*Gate:* `cu ai/generate` (or `cognition/vision-describe`) with an image → a real -description returns from the local model. +**SELF-PROVISIONING, not a manual pull** ([[managed-product-everything-self-provisions-no-operator-steps]]): +a repo user is NOT going to run `huggingface-cli` — the system fetches the model itself. +Concretely: (a) one catalog `Model` row (id, `Vision` capability, `hf_source`/`gguf_hint`, +mmproj path); (b) make the VL bring-up SELF-PROVISION via the EXISTING provisioning path +(`provisioning::Downloader` / `fetch.rs` / `commands/models/pull.rs`) — pull the GGUF **and +its mmproj** from `hf_source`, no CLI dependency; (c) **uniform resolution** — `mmproj` +must resolve local→HF-cache→pull the SAME way the GGUF already does (today +`resolve_mmproj_for_model` only checks the declared local path — the gap that would force a +manual placement); (d) llama-server serves it (`--mmproj` already wired, #1955). Nothing +perceives without a describer, AND nothing ships if a user has to fetch it by hand. +*Gate:* on a clean machine, declaring the VL row + starting the system → the model +auto-provisions → `cu cognition/vision-describe` with an image returns a real description, +**with zero manual download/placement steps.** **Phase 2 — Frame ingest: LiveKit video → `MediaFrame`.** Tap the existing per-participant `video_rx`; wrap sampled frames as content-hash From 5fa10bf92f8a9a7964adb22da89140b8ea46692d Mon Sep 17 00:00:00 2001 From: Joel Teply Date: Sat, 18 Jul 2026 17:05:10 -0500 Subject: [PATCH 5/5] =?UTF-8?q?docs(plan):=20perception=20is=20NON-BLOCKIN?= =?UTF-8?q?G=20+=20a=20MANAGED=20RAG-budget=20source=20=E2=80=94=20async/b?= =?UTF-8?q?uffer-first,=20must=20not=20dominate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Joel: thumbnails/descriptions arrive when they arrive, ABSOLUTELY non-blocking, cached; but they pile up, so like any positronic system they're a managed RAG budget — must not dominate (deep thoughts can't crowd rapid response). Design async/buffer-first, not blocking-then-patched. Baked into the resolution/budget section: (1) cells bridge in when ready, persona never waits; (2) a MediaPerceptionSource:RagSource delivers the room-as-now percepts under the ONE flexbox budget, decayed/focus-weighted, competing with engram/airc/roster, never crowding them. Refs #190, #192, #8, #167. [[command-async-shape-prefer-stream-never-block]], [[budget-at-assembly-not-clamp-the-prompt]], [[alive-fast-and-deep-three-layer-stack]] Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo --- docs/planning/LIVEKIT-PERCEPTION-PIPELINE.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/planning/LIVEKIT-PERCEPTION-PIPELINE.md b/docs/planning/LIVEKIT-PERCEPTION-PIPELINE.md index dff2ec7d5b..c5e4fcb955 100644 --- a/docs/planning/LIVEKIT-PERCEPTION-PIPELINE.md +++ b/docs/planning/LIVEKIT-PERCEPTION-PIPELINE.md @@ -60,6 +60,22 @@ has to be small by default and reused hard ([[perception-feedback-must-not-blow- once for all." Sampling cadence (not 30fps) + one-cell-per-content-hash + shared KV is what makes the N-persona wall affordable. The ambient look is designed to be nearly free on the second-and-Nth consumer. +- **ABSOLUTELY NON-BLOCKING — cells arrive when they arrive** ([[command-async-shape-prefer-stream-never-block]]). + The persona NEVER waits on a thumbnail or a description; the cognitive turn proceeds on + what's ready NOW and the deeper cells (describe/pose/detect) BRIDGE IN as they resolve — + exactly like deep thoughts don't stall the rapid response ([[alive-fast-and-deep-three-layer-stack]], + [[two-tier-resolution-mesh]]). Design async/buffer-first, not blocking-then-patched: a + frame enters a buffer, its cells compute async on `SharedCompute`, the persona reads the + latest resolved projection. A pending cell is simply absent this tick, present the next. +- **MANAGED RAG BUDGET — perception must NOT dominate.** These pile up (frames × participants + × cells over time), so perception is a BUDGETED RAG SOURCE like any other, not a firehose + into context. A `MediaPerceptionSource: RagSource` delivers the CURRENT projected percepts + under a per-turn budget via the one flexbox allocator ([[budget-at-assembly-not-clamp-the-prompt]], + [[situation-aware-focuser]], #8 converged allocator, #167 focus→per-layer budget) — coalesced + to the room-as-it-is-NOW (latest per participant, not a backlog — [[perceive-the-room-as-it-is-now]]), + decayed/evicted like memory, weighted by focus/situation. It competes for context with + engram/airc/roster; it never crowds them out. Half-assing this = a persona drowning in + stale thumbnails instead of thinking. Don't. ## Current state (what's already real — don't rebuild)