From d998e12e83c096cff9cc34897e3c056e18122dc4 Mon Sep 17 00:00:00 2001
From: Karn <mail@karngyan.com>
Date: Sat, 18 Jul 2026 02:17:15 +0530
Subject: [PATCH 1/3] =?UTF-8?q?docs:=20add=20Phase=202=20=E2=80=94=20Speed?=
 =?UTF-8?q?=20(fewer=20agent=20turns)=20to=20roadmap?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Insert a new phase above evals: speed measured in agent turns and
tokens-per-turn, not wire latency. Pull-shaped only — broad push
pre-analysis across tabs is explicitly rejected (debugger conflicts,
wasted work, permission-tier violations), as is a REPL mode.

Ship-now list in win-per-effort order: error responses that embed the
next observation, guarded chains (reins run with per-step assertions),
SKILL.md turn-frugal patterns, snapshot epochs + deltas + structural
compression, reins observe + anticipatory act payloads, declarative
fill-form, sticky tab / digests / screenshot economics. Phase-later:
paginate, self-healing refs, per-site recipe cache, closed-vocabulary
semantic hints.

Renumber subsequent phases (evals v0.5, depth v0.6, reach v0.7+) and
move the old Phase 4 batch-mode bullet into Phase 2 as guarded chains.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---
 docs/ROADMAP.md | 115 ++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 106 insertions(+), 9 deletions(-)

diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md
index c3771cd..ee7956d 100644
--- a/docs/ROADMAP.md
+++ b/docs/ROADMAP.md
@@ -1,6 +1,6 @@
 # reins roadmap
 
-Date: 2026-07-11 (updated; first written 2026-07-08). Living document —
+Date: 2026-07-18 (updated; first written 2026-07-08). Living document —
 reorder freely; phases are priority order, not a calendar.
 
 ## Where reins stands
@@ -28,7 +28,7 @@ reorder freely; phases are priority order, not a calendar.
   trail landed as `reins audit` (#20), and the written threat model
   (docs/SECURITY.md) plus prompt-injection guidance in the skill completed
   the containment story Claude in Chrome set user expectations for. Phase 1
-  is done; the focus shifts to proof (evals) and growth.
+  is done; the focus shifts to speed, proof (evals), and growth.
 
 ## Phase 1 — Trust: a permission model (v0.3) — shipped
 
@@ -59,7 +59,106 @@ the scariest sentence in the README. Ship containment before growth.
   report instruction-shaped text to the user, never move secrets across
   origins.
 
-## Phase 2 — Proof: an eval harness for the skill (v0.4)
+## Phase 2 — Speed: fewer agent turns (v0.4)
+
+The IPC chain (process spawn → daemon → WebSocket → extension → CDP → page)
+is milliseconds; the LLM inference between commands is seconds. So speed is
+measured in **agent turns and tokens-per-turn**, not wire latency — a routine
+flow (find tab, snapshot, click, re-snapshot, verify) is 5–6 full turns
+today. Cutting turns is the only lever that matters; making each turn's
+payload smaller is the second.
+
+Explicitly rejected: broad push/pre-analysis across tabs (the extension
+pre-computing snapshots for pages the agent never asked about). It fights the
+one-debugger-per-tab limit, wastes work on the 38 of 40 tabs the agent never
+touches, and reads sites before the user or agent directed it — quietly
+violating the Phase 1 permission tiers. Also rejected: a long-lived `reins
+repl` — agent harnesses are one-command-per-turn, and everything a REPL would
+buy (session state, diffs) lives correctly in the daemon, which already
+exists. Everything below is pull-shaped and per-driven-tab.
+
+Ship-now, in win-per-effort order:
+
+1. **Every error is an observe.** A failed action returns the context the
+   agent will ask for next: stale ref → snapshot delta since last look;
+   element not found → nearest role/name candidates; dialog open → the dialog
+   text. Failure loops (fail → snapshot → retry = 3 turns) become 1+1.
+   Response-shape change only; the extension has the data at failure time.
+   Rule: a `deny`-tier failure returns zero page-derived context.
+2. **Guarded chains.** `reins run` executes a short action list against one
+   tab, each step carrying optional postconditions (`expect: url~=…`,
+   `visible: <sel>`, `text~=…`, `network: POST /api/x`). Halt at first failed
+   guard and return the full observe-delta at the halt point. Guards are
+   assertions, not branching — that's the line that keeps it deterministic
+   instead of a mini-agent, and what makes batching trustworthy enough to be
+   the default pattern SKILL.md teaches. Routine flow: 5–6 turns → 2;
+   multi-step forms: 8–10 → 2. (Supersedes the Phase 4 `reins run` note.)
+3. **SKILL.md as a zero-code turn compiler.** Teach turn-frugal patterns that
+   need no code: batch independent commands in one shell line (`reins text …
+   && reins network …`) and read both outputs in one turn; check `network`
+   for an API before scraping the DOM (promote the credentialed-`fetch`
+   recipe into the core loop); don't re-snapshot unless the response says the
+   page changed. Update the skill in the same PR as each primitive below so
+   it never teaches stale patterns.
+4. **Snapshot epochs + delta-by-default + structural compression.** Refs
+   anchor to element identity and survive re-render; snapshots are tagged
+   (`snap@3`) and second-and-later snapshots on a driven tab return only the
+   delta (`+e71 button "Confirm" / -e12`) unless `--full`. Epoch resets
+   (navigation) are loud: all prior refs invalid, full snapshot follows.
+   Orthogonally, collapse repeated sibling structures — `e20..e69: 50×
+   listitem {link, price, button "Add"}` with one expanded exemplar, never
+   collapsing rows whose structure differs. 70–95% token cut on the heaviest
+   payloads, and the substrate `observe` is built on.
+5. **`reins observe` + anticipatory payloads.** One compound call answers
+   "what's the page now": url, title, ref delta, key text, new console
+   errors, requests since last look. Then apply one policy to every acting
+   command (`click`/`type`/`nav`/`open`): inline whatever the agent would
+   deterministically request next — small change → delta; navigation → fresh
+   compact snapshot; nothing → `no-op: page unchanged`. Collapses act→look
+   pairs generically, capped payloads, driven-tab only.
+6. **`reins fill-form`.** Declarative form fill: `--data '{"email":…}'`,
+   deterministic key→field matching (autocomplete attr > label > name >
+   placeholder), one pass, per-field match report back. Ambiguous match (2+
+   candidates) = leave unfilled and report, never guess — that rule is the
+   line before "unreliable mini-agent". Never submits without `--submit`.
+   Matching rules published verbatim in SKILL.md. Signup/checkout: 5–8 turns
+   → 1–2.
+7. **Sticky tab + digests + screenshot economics.** `reins use --tab 12`
+   pins subsequent commands to a tab (retires the default-to-active-tab
+   hazard where the user switches tabs mid-task and the agent types into
+   their email; sticky tab closed = loud error, no silent fallback).
+   `console`/`network` return digests (`23 reqs: 18× GET /api/poll (200), 1×
+   POST /api/submit (500)`) with errors always verbatim, `--all` for raw.
+   `screenshot --if-changed` returns `unchanged` instead of a new image;
+   `screenshot --ref e5` crops to the element. Small individually; together
+   they shave the per-turn tax everywhere.
+
+Phase-later, in order: **`reins paginate`** (the one bounded loop worth
+having: click next-control, wait for settle, run an extraction expr per
+page, concatenate JSON, `--max-pages` mandatory; reuses the chain runner —
+but SKILL.md should say "try `network`+`fetch` first, `paginate` is the DOM
+fallback"); **self-healing refs** (stale ref re-matched by role+name only on
+a unique candidate within the same landmark, every heal audit-logged,
+`--strict` to opt out); **per-site recipe cache** (successful chains/fills
+stored as selector maps + guards keyed by exact origin and DOM fingerprint,
+replayed with guards live so a stale recipe degrades to a normal failure —
+safe only if recipes are strictly selectors, never free text, and tier
+enforcement stays extension-side); **semantic page-model hints** (flag the
+primary form / pagination / main content / auth state with the first
+observation). Standing rule adopted now, before hints exist: hint values
+come from a closed vocabulary (`primary-form`, `auth-state:signed-in`, …) —
+the page may influence *which* enum fires, never the words the model reads.
+Free-text hints would be a prompt-injection amplifier; a pre-analyzed page
+is still untrusted web content, and several items above (compression,
+digests) actively *shrink* the injection surface by cutting raw page text.
+
+This phase and Phase 3 (evals) are symbiotic: the metrics here are
+tool-calls-per-task, tokens-per-task, and wall-clock-per-task, and the eval
+harness is how you prove a turn cut actually helped without regressing
+correctness. Land a lightweight per-session tool-call / token counter
+alongside this work even if the full harness comes after.
+
+## Phase 3 — Proof: an eval harness for the skill (v0.5)
 
 No eval suite exists today (unit/integration tests only). Skills are prompts;
 prompts regress silently when models change. Adopt the anthropics/skills
@@ -98,7 +197,7 @@ user bug reports), plus on-demand before each skill edit. Publish the
 scorecard in the README — nobody else in the niche has public skill evals;
 it's both quality control and marketing.
 
-## Phase 3 — Depth: close the capability gaps (v0.5)
+## Phase 4 — Depth: close the capability gaps (v0.6)
 
 Informed by the comparison table; promote by observed demand, not speculation.
 
@@ -106,17 +205,15 @@ Informed by the comparison table; promote by observed demand, not speculation.
   `network` records method/URL/status only — the biggest day-to-day gap.
   HAR-style capture (`reins network --har`, `reins network --body <id>`)
   covers reverse-engineering workflows without the `eval`-fetch detour.
-- **Batch mode.** Chrome DevTools CLI v1 and agent-browser both bet on
-  batching actions per invocation to cut round-trips. A
-  `reins run <script>` (or stdin) executing a short action list against one
-  tab would drop the agent's tool-call count for multi-step flows.
 - **Downloads + PDF** as curated commands (both exist via `cdp` today).
+  (Batch mode moved to Phase 2 as guarded chains — Chrome DevTools CLI v1
+  and agent-browser both bet on batching too; guards are the differentiator.)
 - **WebMCP watch.** Chrome 149 origin trial lets sites expose
   `navigator.modelContext` tools. When it matures, `reins tools <tab>` —
   list and call site-declared tools — turns a structural threat (sites
   bypassing DOM automation) into a feature.
 
-## Phase 4 — Reach: distribution and ecosystem (v0.6+)
+## Phase 5 — Reach: distribution and ecosystem (v0.7+)
 
 - **Discovery.** The funnel is installs-without-visitors; invert it: launch
   post (the CLI-vs-MCP token story + security model is the angle), demo

From d777808412fbcf914592c71f6c25f14594dc1799 Mon Sep 17 00:00:00 2001
From: Karn <mail@karngyan.com>
Date: Sat, 18 Jul 2026 02:31:26 +0530
Subject: [PATCH 2/3] docs: replace sticky tab with stateless explicit-tab rule
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Daemon-side sticky tab breaks under concurrency: several agents drive
several tabs through one daemon, and there is no reliable session
identity to scope a pin by (fresh process per command, no persistent
shell state across harness invocations) — agent A pins, agent B
re-pins, A acts on B's tab.

Fix the underlying hazard statelessly instead: acting commands stop
defaulting to the active tab and error with the roster when more than
one tab is driven, mirroring the existing multiple-browser rule. Read
commands keep the convenience default.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---
 docs/ROADMAP.md | 28 +++++++++++++++++++---------
 1 file changed, 19 insertions(+), 9 deletions(-)

diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md
index ee7956d..7ed0147 100644
--- a/docs/ROADMAP.md
+++ b/docs/ROADMAP.md
@@ -123,15 +123,25 @@ Ship-now, in win-per-effort order:
    line before "unreliable mini-agent". Never submits without `--submit`.
    Matching rules published verbatim in SKILL.md. Signup/checkout: 5–8 turns
    → 1–2.
-7. **Sticky tab + digests + screenshot economics.** `reins use --tab 12`
-   pins subsequent commands to a tab (retires the default-to-active-tab
-   hazard where the user switches tabs mid-task and the agent types into
-   their email; sticky tab closed = loud error, no silent fallback).
-   `console`/`network` return digests (`23 reqs: 18× GET /api/poll (200), 1×
-   POST /api/submit (500)`) with errors always verbatim, `--all` for raw.
-   `screenshot --if-changed` returns `unchanged` instead of a new image;
-   `screenshot --ref e5` crops to the element. Small individually; together
-   they shave the per-turn tax everywhere.
+7. **Explicit tab for acting commands + digests + screenshot economics.**
+   Concurrency is a standing constraint for this whole phase: several agents
+   can drive several tabs through the one daemon at once, so any "current
+   tab" state held daemon-side is cross-agent contamination waiting to
+   happen (agent A pins a tab, agent B re-pins, A acts on B's tab) — and
+   there's no reliable session identity to scope it by, since every command
+   is a fresh process and harnesses don't persist shell state. So no sticky
+   tab. Fix the actual hazard statelessly instead: acting commands
+   (`click` / `type` / `fill` / …) stop defaulting to the active tab — with
+   more than one driven tab they error with the roster (`specify --tab;
+   active is tab 12 "Gmail"`), mirroring the existing multiple-browser rule.
+   Read commands keep the convenience default. That retires the
+   user-switched-tabs-mid-task hazard where the agent types into their
+   email, with zero session state. Separately: `console`/`network` return
+   digests (`23 reqs: 18× GET /api/poll (200), 1× POST /api/submit (500)`)
+   with errors always verbatim, `--all` for raw; `screenshot --if-changed`
+   returns `unchanged` instead of a new image; `screenshot --ref e5` crops
+   to the element. Small individually; together they shave the per-turn tax
+   everywhere.
 
 Phase-later, in order: **`reins paginate`** (the one bounded loop worth
 having: click next-control, wait for settle, run an extraction expr per

From 8581e6814f8f2c6cdccdb15f570b6951a451fed7 Mon Sep 17 00:00:00 2001
From: Karn <mail@karngyan.com>
Date: Sat, 18 Jul 2026 02:39:09 +0530
Subject: [PATCH 3/3] =?UTF-8?q?docs:=20reorder=20roadmap=20=E2=80=94=20eva?=
 =?UTF-8?q?ls=20(Phase=202)=20before=20speed=20(Phase=203)?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Build the ruler before optimizing: the speed phase rewrites the
agent-facing contract (error shapes, snapshot format, the skill's core
loop), and without a baseline there is no way to prove a turn cut
helped or didn't regress correctness. The eval fixture site is also
the testbed the speed primitives are developed against.

Eval phase gains a Baselines bullet (tool-calls / tokens / wall-clock
per task recorded on every run) and a note that trajectory assertions
track whatever SKILL.md currently teaches. The speed phase's shipping
bar becomes: baselines move, execution evals stay green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---
 docs/ROADMAP.md | 112 +++++++++++++++++++++++++++---------------------
 1 file changed, 64 insertions(+), 48 deletions(-)

diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md
index 7ed0147..d6ed120 100644
--- a/docs/ROADMAP.md
+++ b/docs/ROADMAP.md
@@ -28,7 +28,8 @@ reorder freely; phases are priority order, not a calendar.
   trail landed as `reins audit` (#20), and the written threat model
   (docs/SECURITY.md) plus prompt-injection guidance in the skill completed
   the containment story Claude in Chrome set user expectations for. Phase 1
-  is done; the focus shifts to speed, proof (evals), and growth.
+  is done; the focus shifts to proof (evals) first — the ruler — then speed
+  measured against it, then growth.
 
 ## Phase 1 — Trust: a permission model (v0.3) — shipped
 
@@ -59,14 +60,67 @@ the scariest sentence in the README. Ship containment before growth.
   report instruction-shaped text to the user, never move secrets across
   origins.
 
-## Phase 2 — Speed: fewer agent turns (v0.4)
+## Phase 2 — Proof: an eval harness for the skill (v0.4)
+
+Evals come before the speed work deliberately: Phase 3 rewrites the
+agent-facing contract (error shapes, snapshot format, the skill's core
+loop), and without a baseline there is no way to prove a turn cut helped —
+or didn't regress correctness — except user bug reports. The fixture site
+built here is also the testbed the Phase 3 primitives (guarded chains,
+fill-form, paginate) are developed against. Build the ruler, then optimize.
+
+No eval suite exists today (unit/integration tests only). Skills are prompts;
+prompts regress silently when models change. Adopt the anthropics/skills
+`skill-creator` methodology — it measures three separate things:
+
+1. **Triggering** — does the skill load when it should (and not when it
+   shouldn't)? Harness: `evals/trigger-eval.json` with ~10 realistic
+   should-trigger prompts ("scrape my dashboard behind SSO", "watch what API
+   this page calls") and ~10 near-miss should-NOT-trigger prompts (generic web
+   questions, Playwright test authoring, curl-able public APIs). Run each 3×
+   via `claude -p --output-format stream-json` and detect the Skill tool_use
+   event (skill-creator's `run_eval.py` pattern). Metric: precision/recall.
+   Anthropic documents under-triggering as the default failure mode;
+   `run_loop.py` can optimize the description against a train/test split.
+2. **Execution** — once loaded, does the agent drive reins correctly?
+   Browser evals against live sites are flaky, so favor trajectory and
+   artifact assertions over output diffs:
+   - **Fixture site.** A tiny local app (login wall, paginated table, form
+     wizard, JSON API, dialog traps) served by the eval runner —
+     deterministic ground truth.
+   - **Assertions.** Did the transcript call `snapshot` before `click`? Did it
+     re-snapshot after nav? Did it produce the exact JSON the fixture defines?
+     Exit codes and artifacts, not prose grading, wherever possible.
+     Trajectory assertions encode whatever SKILL.md currently teaches — when
+     Phase 3 changes the core loop, the assertions change in the same PR.
+   - **Baselines.** Every run records tool-calls-per-task, tokens-per-task,
+     and wall-clock-per-task. These are the numbers Phase 3's turn-cutting
+     work is measured against, and the regression tripwire once it lands.
+   - The existing `integration.test.ts` (daemon + stand-in WS extension)
+     already fakes the browser end — reuse it for a cheap deterministic tier;
+     run a real-browser tier less often.
+3. **Uplift** — is the skill better than no skill? Paired runs (with-skill vs
+   `without_skill` baseline, fresh sessions, parallel subagents), graded and
+   aggregated to pass-rate / tokens / time, mean ± stddev across repeats
+   (skill-creator's `aggregate_benchmark.py` shape). Variance analysis
+   separates flaky evals from real regressions.
+
+Operationalize: `evals/` directory in-repo, `pnpm eval` runner, weekly CI run
+with pinned model IDs (silent model updates should surface as eval diffs, not
+user bug reports), plus on-demand before each skill edit. Publish the
+scorecard in the README — nobody else in the niche has public skill evals;
+it's both quality control and marketing.
+
+## Phase 3 — Speed: fewer agent turns (v0.5)
 
 The IPC chain (process spawn → daemon → WebSocket → extension → CDP → page)
 is milliseconds; the LLM inference between commands is seconds. So speed is
 measured in **agent turns and tokens-per-turn**, not wire latency — a routine
 flow (find tab, snapshot, click, re-snapshot, verify) is 5–6 full turns
 today. Cutting turns is the only lever that matters; making each turn's
-payload smaller is the second.
+payload smaller is the second. Phase 2's harness is the precondition: every
+item below lands measured against the baseline, with the execution evals as
+the regression tripwire.
 
 Explicitly rejected: broad push/pre-analysis across tabs (the extension
 pre-computing snapshots for pages the agent never asked about). It fights the
@@ -162,50 +216,12 @@ Free-text hints would be a prompt-injection amplifier; a pre-analyzed page
 is still untrusted web content, and several items above (compression,
 digests) actively *shrink* the injection surface by cutting raw page text.
 
-This phase and Phase 3 (evals) are symbiotic: the metrics here are
-tool-calls-per-task, tokens-per-task, and wall-clock-per-task, and the eval
-harness is how you prove a turn cut actually helped without regressing
-correctness. Land a lightweight per-session tool-call / token counter
-alongside this work even if the full harness comes after.
-
-## Phase 3 — Proof: an eval harness for the skill (v0.5)
-
-No eval suite exists today (unit/integration tests only). Skills are prompts;
-prompts regress silently when models change. Adopt the anthropics/skills
-`skill-creator` methodology — it measures three separate things:
-
-1. **Triggering** — does the skill load when it should (and not when it
-   shouldn't)? Harness: `evals/trigger-eval.json` with ~10 realistic
-   should-trigger prompts ("scrape my dashboard behind SSO", "watch what API
-   this page calls") and ~10 near-miss should-NOT-trigger prompts (generic web
-   questions, Playwright test authoring, curl-able public APIs). Run each 3×
-   via `claude -p --output-format stream-json` and detect the Skill tool_use
-   event (skill-creator's `run_eval.py` pattern). Metric: precision/recall.
-   Anthropic documents under-triggering as the default failure mode;
-   `run_loop.py` can optimize the description against a train/test split.
-2. **Execution** — once loaded, does the agent drive reins correctly?
-   Browser evals against live sites are flaky, so favor trajectory and
-   artifact assertions over output diffs:
-   - **Fixture site.** A tiny local app (login wall, paginated table, form
-     wizard, JSON API, dialog traps) served by the eval runner —
-     deterministic ground truth.
-   - **Assertions.** Did the transcript call `snapshot` before `click`? Did it
-     re-snapshot after nav? Did it produce the exact JSON the fixture defines?
-     Exit codes and artifacts, not prose grading, wherever possible.
-   - The existing `integration.test.ts` (daemon + stand-in WS extension)
-     already fakes the browser end — reuse it for a cheap deterministic tier;
-     run a real-browser tier less often.
-3. **Uplift** — is the skill better than no skill? Paired runs (with-skill vs
-   `without_skill` baseline, fresh sessions, parallel subagents), graded and
-   aggregated to pass-rate / tokens / time, mean ± stddev across repeats
-   (skill-creator's `aggregate_benchmark.py` shape). Variance analysis
-   separates flaky evals from real regressions.
-
-Operationalize: `evals/` directory in-repo, `pnpm eval` runner, weekly CI run
-with pinned model IDs (silent model updates should surface as eval diffs, not
-user bug reports), plus on-demand before each skill edit. Publish the
-scorecard in the README — nobody else in the niche has public skill evals;
-it's both quality control and marketing.
+The bar for shipping anything in this phase: the Phase 2 baselines
+(tool-calls / tokens / wall-clock per task) move in the right direction and
+the execution evals stay green. A speed item that doesn't move the numbers —
+or buys them by regressing correctness — doesn't ship. Separately, a
+lightweight per-session tool-call counter (local only, no telemetry) is
+still worth landing for real-world sessions the harness can't see.
 
 ## Phase 4 — Depth: close the capability gaps (v0.6)
 
@@ -216,7 +232,7 @@ Informed by the comparison table; promote by observed demand, not speculation.
   HAR-style capture (`reins network --har`, `reins network --body <id>`)
   covers reverse-engineering workflows without the `eval`-fetch detour.
 - **Downloads + PDF** as curated commands (both exist via `cdp` today).
-  (Batch mode moved to Phase 2 as guarded chains — Chrome DevTools CLI v1
+  (Batch mode moved to Phase 3 as guarded chains — Chrome DevTools CLI v1
   and agent-browser both bet on batching too; guards are the differentiator.)
 - **WebMCP watch.** Chrome 149 origin trial lets sites expose
   `navigator.modelContext` tools. When it matures, `reins tools <tab>` —