Voice co-pilot, LLM agents, genie world-authoring, and a procedural score

[dusterbloom]

Adds three agent systems, a creative world-authoring surface, and a rewritten soundtrack. Everything runs in the page — no server of ours in the loop, and no paid API anywhere in the UI.

Talk to the carpet

A co-pilot you speak or type at. Each turn returns a spoken reply plus an intent it acts on, grounded in agentAPI.observe() so it talks about the world you're actually in: take me to the crystal formation, raise a mountain ahead, make it sunset, let me fly the flamingo.

Three seams, each a factory returning the same shape, so a fourth backend never touches the caller:

• llmProviders.js — any OpenAI-compatible endpoint, or WebLLM on WebGPU
• voiceProviders.js — the OS voice (default), Kokoro, or Supertonic-3
• listenModes.js — push-to-talk, or hands-free with semantic endpointing

localEndpoints.js probes the usual loopback ports, so anyone already running a local server is offered it before being asked for anything.

Every brain is free

The Brain picker offers Groq, OpenRouter (filtered to :free models), a local server, on-device WebLLM, and Apple FM. Anthropic is gone from the UI; its provider code, modelRouting.js and llmBudget.js stay on disk so an existing saved config keeps working.

Model ids are fetched live from each provider's /models rather than hardcoded, because hardcoded ids rot. probeEndpoint() already did that fetch for the local scan and now serves both callers.

Keys are scoped per brain, not per provider. Groq and OpenRouter both resolve to the openai provider at runtime, so the previous provider-keyed map would have sent a Groq key to OpenRouter — exactly what that map's comment says it exists to prevent.

The picker also learns. Some free models are gated to "agentic harnesses" and 403 from an app like this, and nothing in the /models metadata says so — pricing is zero, per_request_limits is null. So a 403 naming the model in use blocklists it for that brain, advances to the next candidate, and says so once. Requiring the model id in the error text is what keeps a bad API key — also a 403 — from quietly emptying the dropdown.

Apple FM

fm serve (macOS 26+) puts an OpenAI-compatible server on port 1976 — no key, no download, already there. It's probed second, behind only this project's own server.

Measured on a real Mac against the game's actual system prompt, 24 turns each:

model	parseable	correct	median	where it runs
pcc	24/24	24/24	1650ms	Apple's Private Cloud Compute
system	24/24	13–17/24 (unstable)	3400ms	your Mac

pcc is the default because the data says so, and the README says plainly that it is a network call, not on-device — only system is. Two fixes made it work: stream: false (fm serve streams SSE otherwise, which res.json() can't parse), and preferModels, because Apple lists system first so models[0] would have defaulted to the weaker model by accident of array order.

Conversation surface

CopilotConsole is a persistent chat panel. Before it, every turn — your words, the reply, the world edits — rendered as a toast and vanished, and there was no text input anywhere in the UI: a browser without SpeechRecognition had no path in but DevTools. Voice and typing are now two inputs to one conversation, and Firefox works.

UI

Tabs reorder to Agent / Flight / Race with Agent first. The Agent tab is one click and talk — Talk, Voice Off, and planet save/load, with settings collapsed once a brain is configured. SimpleBot and the LLM pilot lose their buttons but keep their code and their window.agentAPI entry points. A music mute sits in the HUD, one tap, no menu.

Genie world-authoring

window.worldAPI, deliberately separate from the fairness-bound agentAPI so creative edits can't taint a verified race. Conjure primitives and animated rigs, import from curated CORS-friendly repos, fly what you import, roam the ground as an animal, reshape terrain and sky, save and restore a planet. GenieCatalog persists it to IndexedDB.

Score

ProceduralMusicSystem rewritten around an actual composition: a theme derived from the planet seed, real chord progressions, a nine-phrase song form, and Karplus-Strong strings through a generated impulse-response reverb. Reroll the world and the music changes with it.

Found by playing it

Five bugs the automated tests could not have caught, all surfaced by driving the co-pilot against a real model:

• World ops were started in order, not applied in order. _applyWorld looped a synchronous _applyWorldOp that fired async verbs without returning their promises. The exact sequence the system prompt documents — [{import Fox},{vehicle Fox}] — kicked off the import's fetch then immediately ran the vehicle lookup against an empty catalogue. Every "let me fly the X" left the model in the world and the player on the carpet.
• A resolved-but-falsy result was reported as success. vehicle() returns false on a miss and spawn() returns [], but the bridge treated resolution as success — so the console said "🪄 now flying the Fox" while nothing had happened. That is why the failure was silent, and fixing it is what made every later diagnosis possible.
• Catalogue names were looked up exactly, but stored slugified. uniqueName() writes "Flamingo" as flamingo; get() is a case-sensitive Map lookup. Every capitalised name missed, and the model cannot know the slug because it is generated after the model has written its ops. resolve() now slugifies with the same shared rule, matches exact-family → startsWith → includes without ever falling to a looser tier, and prefers the newest of a collision family.
• Truncated JSON was read aloud. A cut-off reply failed extractJSON, and the fallback put the raw object into the chat bubble and the TTS. It is now repaired where recoverable — the reported string recovers its reply and its complete import op — and never spoken raw.
• The prompt named only one import repo. threejs (the animated birds) and cesium existed in code but were invisible to the model, so a flamingo was unreachable by voice.

Testing

Build, build:pages and smoke pass; the game boots headless with no page errors. Voice turns driven end-to-end against a stub OpenAI-compatible endpoint. Mic behaviour driven through a faked SpeechRecognition. Apple FM discovery and model preference driven against a stub shaped like fm serve. Layout measured at 390×844 and 1280×800. The pure functions — endpointing, voice ranking, model preference, catalogue resolution, JSON repair — were each checked against case tables, and the ordering and false-success bugs were demonstrated before and after against the same script.

Not verified anywhere: real audio output, a real microphone, and a real fm serve process. The stubs cover the wiring, not the hardware.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YYuB2vmEkWjXUWe6HzF8T1