docs: roadmap — evals first (v0.4), then Speed: fewer agent turns (v0.5)

docs/ROADMAP.md

@@ -1,6 +1,6 @@
 # reins roadmap
 
-Date: 2026-07-11 (updated; first written 2026-07-08). Living document —
+Date: 2026-07-18 (updated; first written 2026-07-08). Living document —
 reorder freely; phases are priority order, not a calendar.
 
 ## Where reins stands
@@ -28,7 +28,8 @@ reorder freely; phases are priority order, not a calendar.
   trail landed as `reins audit` (#20), and the written threat model
   (docs/SECURITY.md) plus prompt-injection guidance in the skill completed
   the containment story Claude in Chrome set user expectations for. Phase 1
-  is done; the focus shifts to proof (evals) and growth.
+  is done; the focus shifts to proof (evals) first — the ruler — then speed
+  measured against it, then growth.
 
 ## Phase 1 — Trust: a permission model (v0.3) — shipped
 
@@ -61,6 +62,13 @@ the scariest sentence in the README. Ship containment before growth.
 
 ## Phase 2 — Proof: an eval harness for the skill (v0.4)
 
+Evals come before the speed work deliberately: Phase 3 rewrites the
+agent-facing contract (error shapes, snapshot format, the skill's core
+loop), and without a baseline there is no way to prove a turn cut helped —
+or didn't regress correctness — except user bug reports. The fixture site
+built here is also the testbed the Phase 3 primitives (guarded chains,
+fill-form, paginate) are developed against. Build the ruler, then optimize.
+
 No eval suite exists today (unit/integration tests only). Skills are prompts;
 prompts regress silently when models change. Adopt the anthropics/skills
 `skill-creator` methodology — it measures three separate things:
@@ -83,6 +91,11 @@ prompts regress silently when models change. Adopt the anthropics/skills
    - **Assertions.** Did the transcript call `snapshot` before `click`? Did it
      re-snapshot after nav? Did it produce the exact JSON the fixture defines?
      Exit codes and artifacts, not prose grading, wherever possible.
+     Trajectory assertions encode whatever SKILL.md currently teaches — when
+     Phase 3 changes the core loop, the assertions change in the same PR.
+   - **Baselines.** Every run records tool-calls-per-task, tokens-per-task,
+     and wall-clock-per-task. These are the numbers Phase 3's turn-cutting
+     work is measured against, and the regression tripwire once it lands.
    - The existing `integration.test.ts` (daemon + stand-in WS extension)
      already fakes the browser end — reuse it for a cheap deterministic tier;
      run a real-browser tier less often.
@@ -98,25 +111,135 @@ user bug reports), plus on-demand before each skill edit. Publish the
 scorecard in the README — nobody else in the niche has public skill evals;
 it's both quality control and marketing.
 
-## Phase 3 — Depth: close the capability gaps (v0.5)
+## Phase 3 — Speed: fewer agent turns (v0.5)
+
+The IPC chain (process spawn → daemon → WebSocket → extension → CDP → page)
+is milliseconds; the LLM inference between commands is seconds. So speed is
+measured in **agent turns and tokens-per-turn**, not wire latency — a routine
+flow (find tab, snapshot, click, re-snapshot, verify) is 5–6 full turns
+today. Cutting turns is the only lever that matters; making each turn's
+payload smaller is the second. Phase 2's harness is the precondition: every
+item below lands measured against the baseline, with the execution evals as
+the regression tripwire.
+
+Explicitly rejected: broad push/pre-analysis across tabs (the extension
+pre-computing snapshots for pages the agent never asked about). It fights the
+one-debugger-per-tab limit, wastes work on the 38 of 40 tabs the agent never
+touches, and reads sites before the user or agent directed it — quietly
+violating the Phase 1 permission tiers. Also rejected: a long-lived `reins
+repl` — agent harnesses are one-command-per-turn, and everything a REPL would
+buy (session state, diffs) lives correctly in the daemon, which already
+exists. Everything below is pull-shaped and per-driven-tab.
+
+Ship-now, in win-per-effort order:
+
+1. **Every error is an observe.** A failed action returns the context the
+   agent will ask for next: stale ref → snapshot delta since last look;
+   element not found → nearest role/name candidates; dialog open → the dialog
+   text. Failure loops (fail → snapshot → retry = 3 turns) become 1+1.
+   Response-shape change only; the extension has the data at failure time.
+   Rule: a `deny`-tier failure returns zero page-derived context.
+2. **Guarded chains.** `reins run` executes a short action list against one
+   tab, each step carrying optional postconditions (`expect: url~=…`,
+   `visible: <sel>`, `text~=…`, `network: POST /api/x`). Halt at first failed
+   guard and return the full observe-delta at the halt point. Guards are
+   assertions, not branching — that's the line that keeps it deterministic
+   instead of a mini-agent, and what makes batching trustworthy enough to be
+   the default pattern SKILL.md teaches. Routine flow: 5–6 turns → 2;
+   multi-step forms: 8–10 → 2. (Supersedes the Phase 4 `reins run` note.)
+3. **SKILL.md as a zero-code turn compiler.** Teach turn-frugal patterns that
+   need no code: batch independent commands in one shell line (`reins text …
+   && reins network …`) and read both outputs in one turn; check `network`
+   for an API before scraping the DOM (promote the credentialed-`fetch`
+   recipe into the core loop); don't re-snapshot unless the response says the
+   page changed. Update the skill in the same PR as each primitive below so
+   it never teaches stale patterns.
+4. **Snapshot epochs + delta-by-default + structural compression.** Refs
+   anchor to element identity and survive re-render; snapshots are tagged
+   (`snap@3`) and second-and-later snapshots on a driven tab return only the
+   delta (`+e71 button "Confirm" / -e12`) unless `--full`. Epoch resets
+   (navigation) are loud: all prior refs invalid, full snapshot follows.
+   Orthogonally, collapse repeated sibling structures — `e20..e69: 50×
+   listitem {link, price, button "Add"}` with one expanded exemplar, never
+   collapsing rows whose structure differs. 70–95% token cut on the heaviest
+   payloads, and the substrate `observe` is built on.
+5. **`reins observe` + anticipatory payloads.** One compound call answers
+   "what's the page now": url, title, ref delta, key text, new console
+   errors, requests since last look. Then apply one policy to every acting
+   command (`click`/`type`/`nav`/`open`): inline whatever the agent would
+   deterministically request next — small change → delta; navigation → fresh
+   compact snapshot; nothing → `no-op: page unchanged`. Collapses act→look
+   pairs generically, capped payloads, driven-tab only.
+6. **`reins fill-form`.** Declarative form fill: `--data '{"email":…}'`,
+   deterministic key→field matching (autocomplete attr > label > name >
+   placeholder), one pass, per-field match report back. Ambiguous match (2+
+   candidates) = leave unfilled and report, never guess — that rule is the
+   line before "unreliable mini-agent". Never submits without `--submit`.
+   Matching rules published verbatim in SKILL.md. Signup/checkout: 5–8 turns
+   → 1–2.
+7. **Explicit tab for acting commands + digests + screenshot economics.**
+   Concurrency is a standing constraint for this whole phase: several agents
+   can drive several tabs through the one daemon at once, so any "current
+   tab" state held daemon-side is cross-agent contamination waiting to
+   happen (agent A pins a tab, agent B re-pins, A acts on B's tab) — and
+   there's no reliable session identity to scope it by, since every command
+   is a fresh process and harnesses don't persist shell state. So no sticky
+   tab. Fix the actual hazard statelessly instead: acting commands
+   (`click` / `type` / `fill` / …) stop defaulting to the active tab — with
+   more than one driven tab they error with the roster (`specify --tab;
+   active is tab 12 "Gmail"`), mirroring the existing multiple-browser rule.
+   Read commands keep the convenience default. That retires the
+   user-switched-tabs-mid-task hazard where the agent types into their
+   email, with zero session state. Separately: `console`/`network` return
+   digests (`23 reqs: 18× GET /api/poll (200), 1× POST /api/submit (500)`)
+   with errors always verbatim, `--all` for raw; `screenshot --if-changed`
+   returns `unchanged` instead of a new image; `screenshot --ref e5` crops
+   to the element. Small individually; together they shave the per-turn tax
+   everywhere.
+
+Phase-later, in order: **`reins paginate`** (the one bounded loop worth
+having: click next-control, wait for settle, run an extraction expr per
+page, concatenate JSON, `--max-pages` mandatory; reuses the chain runner —
+but SKILL.md should say "try `network`+`fetch` first, `paginate` is the DOM
+fallback"); **self-healing refs** (stale ref re-matched by role+name only on
+a unique candidate within the same landmark, every heal audit-logged,
+`--strict` to opt out); **per-site recipe cache** (successful chains/fills
+stored as selector maps + guards keyed by exact origin and DOM fingerprint,
+replayed with guards live so a stale recipe degrades to a normal failure —
+safe only if recipes are strictly selectors, never free text, and tier
+enforcement stays extension-side); **semantic page-model hints** (flag the
+primary form / pagination / main content / auth state with the first
+observation). Standing rule adopted now, before hints exist: hint values
+come from a closed vocabulary (`primary-form`, `auth-state:signed-in`, …) —
+the page may influence *which* enum fires, never the words the model reads.
+Free-text hints would be a prompt-injection amplifier; a pre-analyzed page
+is still untrusted web content, and several items above (compression,
+digests) actively *shrink* the injection surface by cutting raw page text.
+
+The bar for shipping anything in this phase: the Phase 2 baselines
+(tool-calls / tokens / wall-clock per task) move in the right direction and
+the execution evals stay green. A speed item that doesn't move the numbers —
+or buys them by regressing correctness — doesn't ship. Separately, a
+lightweight per-session tool-call counter (local only, no telemetry) is
+still worth landing for real-world sessions the harness can't see.
+
+## Phase 4 — Depth: close the capability gaps (v0.6)
 
 Informed by the comparison table; promote by observed demand, not speculation.
 
 - **Network bodies and headers.** The skill's own gotchas section admits
   `network` records method/URL/status only — the biggest day-to-day gap.
   HAR-style capture (`reins network --har`, `reins network --body <id>`)
   covers reverse-engineering workflows without the `eval`-fetch detour.
-- **Batch mode.** Chrome DevTools CLI v1 and agent-browser both bet on
-  batching actions per invocation to cut round-trips. A
-  `reins run <script>` (or stdin) executing a short action list against one
-  tab would drop the agent's tool-call count for multi-step flows.
 - **Downloads + PDF** as curated commands (both exist via `cdp` today).
+  (Batch mode moved to Phase 3 as guarded chains — Chrome DevTools CLI v1
+  and agent-browser both bet on batching too; guards are the differentiator.)
 - **WebMCP watch.** Chrome 149 origin trial lets sites expose
   `navigator.modelContext` tools. When it matures, `reins tools <tab>` —
   list and call site-declared tools — turns a structural threat (sites
   bypassing DOM automation) into a feature.
 
-## Phase 4 — Reach: distribution and ecosystem (v0.6+)
+## Phase 5 — Reach: distribution and ecosystem (v0.7+)
 
 - **Discovery.** The funnel is installs-without-visitors; invert it: launch
   post (the CLI-vs-MCP token story + security model is the angle), demo