diff --git a/docs/loop-review-patterns.md b/docs/loop-review-patterns.md index 06d6c53..69f538b 100644 --- a/docs/loop-review-patterns.md +++ b/docs/loop-review-patterns.md @@ -22,6 +22,12 @@ irrelevant blockers or declare itself done on vibes. | Reviewer | LLM judgment | scoped to the structural/architectural residual specs can't capture. | | Human | closure authority | HITL sign-off for subjective or high-stakes decisions. | +Optional **typed decision layer** (System One / Jev — proposed): sits beside the +generative reviewer, never replaces it. After green verify, a cheap decisions API +answers fixed-schema questions (`noul` / `choice` / `score`) over fenced state; +it does not write `review.md` or scaffold GOAL/verify. See pattern 9 and +[`system-one-review.md`](./system-one-review.md). + Key warning (Specification as Quality Gate, 2026): an LLM reviewing LLM-generated code is **circular** without an external reference — both share a training distribution and echo each other's errors. So `verify` is the real gate; @@ -109,6 +115,34 @@ separate cross-loop reviewer design, validated. Their taxonomy also reminds us: **start simple, use patterns selectively** — `reviewGate` off is the right path for trivial tasks; the full gating stack is for high-stakes work. +### 9. Typed decision layer (System One / Jev) beside generative review +LangChain and TypeSafe have experimented with **Jev** as a judge: same rubric, +typed outputs, and probability scores instead of free-form critique — useful for +**consistency and cost** on repeatable residual checks. Agent Looper maps that to +an optional layer **after verify**, not a new `reviewRuntime`: + +1. **Verifier** — unchanged hard gate. +2. **Generative reviewer** — `reviewRuntime` / `reviewModel` (and optional + `reviewSecondaryRuntime`) still produce `review.md`, Guide packets, and + impact-severity blockers; `agent-loop-prompt` still uses this judge to *write* + GOAL + verify. +3. **System One** — one OpenRouter decisions call (`typesafe/jev-1.13` or + `~typesafe/jev-latest`) with harness-fixed questions derived from REVIEWS + residual defaults (e.g. noul `does_pass`, score `residual_quality`, choice + `blocker_class`). + +Jev does **not** generate text. Wiring it as a chat judge would break scaffold +and `reviewGate` prose contracts. Instead, `systemOne.gate` controls grind impact: +`advisory` surfaces typed answers in `run-report.md` (fail open on API errors); +`block` may reopen the loop on explicit thresholds (fail closed on API/parse +errors). Typed layer may run parallel to or after generative review; it never +replaces verify or human closure (`reviewGateHitl`). + +Maps to us: complements M3 multi-family **generative** secondary judge — secondary +still merges markdown blockers; System One adds a calibrated sensor for ops +metrics and optional cheap gating. Not on `costPreset` minmax. Detail: +[`system-one-review.md`](./system-one-review.md); roadmap M11. + ## The meta-loop (your "10 loops' diffs → review") Scaling to a software factory is **not** bolting more gating onto every loop. It is @@ -132,10 +166,9 @@ blueprint for building it. the right line, (c) did it hallucinate a blocker. That's how the 75% false-blocker inflation was discovered — and the only way to know if `reviewGate` helps or thrashes. -- **Highest-ROI next step:** add an **impact-severity contract** to blockers in - `reviewVerdict` (gate only on `error`-with-impact; downgrade the rest to advisory). - It is the single change the 2026 data says most directly stops the irrelevant- - blocker fix-loop. Full build order, acceptance criteria, and Taskwarrior mapping: +- **Highest-ROI next step (shipped M1):** impact-severity contract in + `reviewVerdict`. Next optional sensor: typed System One layer (roadmap M11) — + not a substitute for generative `reviewGate`. Full build order: [`loop-review-roadmap.md`](./loop-review-roadmap.md). - **Encode misses as system improvements:** when a loop result doesn't meet standard, turn the fix into a reusable skill/check (pattern 6) or a diff --git a/docs/loop-review-roadmap.md b/docs/loop-review-roadmap.md index f66f24f..0e567bf 100644 --- a/docs/loop-review-roadmap.md +++ b/docs/loop-review-roadmap.md @@ -12,9 +12,13 @@ Concrete implementation plan for the upgrades called out in [`loop-review-patterns.md`](./loop-review-patterns.md). Principles stay fixed: 1. **Verifier** is the hard gate (`verify` / `finalVerify`). -2. **Reviewer** is gated judgment on top (never replaces verify). +2. **Reviewer** is gated judgment on top (never replaces verify) — generative + `reviewRuntime` / `reviewModel` (+ optional `reviewSecondaryRuntime`). 3. **Human** remains closure authority for residual / high-stakes calls (`reviewGateHitl`). +Optional **typed decision layer** (`systemOne` / Jev) is documented as M11; it is +not a fourth principle and not a `reviewRuntime` substitute. + Research context and citations live in the patterns doc. This file is the **build order, acceptance criteria, and file touch list**. @@ -52,6 +56,7 @@ Open Taskwarrior project for **Agent Looper** (slug **`agent-loop`**). Use **UUI | M3 | `adf66bf8-d52a-43e2-8009-756649cc32b2` | Multi-family secondary judge (shipped) | | M4 | `fe3f4076-b997-4d28-a59a-baf720c28e5d` | Verification-as-skill | | M5 | `06dec3c5-b35d-4e8a-bb95-c0f2a9ae4f00` | Cross-loop meta-review CLI | +| M11 (proposed) | *(no UUID yet)* | System One / Jev typed review layer | Other Agent Looper backlog: `17bfc1cd-bf5d-43a7-9b8b-9bf7658aaa07` (extract review-gate) — **done**, `de4144f2-9e6a-4cf6-8943-81efc49d4c5c` (loopRisk profiles) — **shipped**, @@ -69,6 +74,7 @@ Other Agent Looper backlog: `17bfc1cd-bf5d-43a7-9b8b-9bf7658aaa07` (extract revi | 3 | Multi-provider / multi-family review | Medium–high (bias reduction) | L | #1 useful first | | 4 | Verification-as-skill | High for quality, orthogonal | M | — | | 5 | Cross-loop meta-reviewer | Factory scale | L | #1–2 data shape helps | +| 6 | System One / Jev typed layer | Medium (cost + consistency sensor) | M | Orthogonal to #1–3; needs generative review contract stable | Pilot rule (from patterns): tune `maxReviewCycles` / `unparseableReviewRetries` on a handful of real loops before locking defaults for #1–3. @@ -342,6 +348,99 @@ much better; not a hard code dependency. --- +## 6. System One / Jev typed review layer + +### Status + +**Proposed** — design + docs only; no harness implementation on the M11 doc PR. +Taskwarrior UUID pending. + +### Problem + +Generative judges are the right tool for `review.md`, Guide packets, and +`agent-loop-prompt` scaffold prose, but they are expensive and noisy for +**repeatable residual rubrics** (pass/fail, quality score, blocker class). Typed +decision models (TypeSafe Jev via OpenRouter System One) answer fixed questions +with probabilities — useful as a **sensor**, not as a replacement for verify or +markdown review. + +### Design + +**Not** `reviewRuntime`. Separate `loop.json` block: + +```json +{ + "reviewGate": true, + "reviewRuntime": "cursor", + "reviewModel": "grok-4.6-high", + "systemOne": { + "enabled": true, + "provider": "openrouter", + "model": "typesafe/jev-1.13", + "gate": "advisory" + } +} +``` + +| `systemOne.gate` | Behavior | +| --- | --- | +| `off` | Disabled (default when block omitted). | +| `advisory` | Log + `run-report.md`; API/parse errors **fail open**; do not reopen worker on typed alone unless product adds explicit thresholds. | +| `block` | Configured thresholds may reopen fix loop; API/parse errors **fail closed** (same seriousness as `reviewGate` infrastructure failures). | + +**API:** OpenRouter `POST …/api/alpha/decisions` with `model`, `state`, `questions` +— not chat completions. Models: `typesafe/jev-1.13`, alias `~typesafe/jev-latest`. + +**State:** fenced bundle (GOAL excerpt, diff summary, `REVIEWS.md`, optional +truncated `review.md`). + +**Questions:** harness-fixed schema from residual defaults — examples: noul +`does_pass`, score `residual_quality`, choice `blocker_class`. One call per +review cycle after green verify. + +**Ordering:** after verify green; may run parallel to or after generative primary/ +secondary review. Shipped `reviewSecondaryRuntime` stays generative (markdown +merge); System One does not author `review.md`. + +**Auth:** existing `OPENROUTER_API_KEY` BYOK path ([`opencode-providers.md`](./opencode-providers.md)). + +Full API notes: [`system-one-review.md`](./system-one-review.md). Pattern motivation: +[`loop-review-patterns.md`](./loop-review-patterns.md) §9. + +### Implementation sketch + +| Area | Work | +| --- | --- | +| `loopConfig.ts` | `systemOne` zod object; default disabled | +| `loopPostReview.ts` | Invoke decisions client after verify; respect `gate` | +| New `src/review/systemOne*.ts` | OpenRouter decisions client, question templates, threshold eval | +| `run-report.md` | Advisory fields + block reasons | +| `failure-domains.ndjson` | Optional `system_one_*` reasons | +| Catalog follow-up | Separate PR to `pnpm sync:models` when models.dev lists Jev | + +### Acceptance criteria + +- [ ] With `systemOne.enabled: false` (or unset), behavior identical to today. +- [ ] Generative `reviewGate` + secondary merge unchanged when System One off. +- [ ] `gate: advisory` never blocks completion on API/parse failure. +- [ ] `gate: block` fails closed on API/parse failure. +- [ ] No code path treats Jev as `reviewRuntime` or `agent-loop-prompt` judge. +- [ ] Docs + dogfood loop example; model ids exactly `typesafe/jev-1.13` / + `~typesafe/jev-latest`. + +### Non-goals + +- `costPreset` minmax inclusion. +- Scaffold or `review.md` authorship. +- Chat-slug `reviewModel` for Jev. +- Replacing impact-severity or reproduce-before-report. + +### Depends on + +Soft: stable generative review pipeline (#1–3 shipped). Hard: none. + +--- + ## Validation experiment (shared) Before declaring #1–3 “done”, run a small offline experiment (patterns doc): @@ -370,6 +469,7 @@ PR that lands #1. | **M8** | Batch rubrics / Auto model (optional) | **Shipped** — batch `{path,rubric}`; computer-use templates; Auto **blocked** pending SDK (`docs/cursor-auto-router.md`) | | **M9** | Linear Loops governance steals | **Shipped (docs/templates)** — prove→freeze, draft discipline, run-report as audit surface, `LOOP.permissions.example.md`, tool default-deny. No harness publish-snapshot yet | | **M10** | Lunar graph scoreboard | **0.4.5** — docs (edge sentences + GOAL/preflight spec fields) + `run-report.md` report card (phase time, kill rate, retries, HITL, writer vs referee $). Hung-worker escalate + `check-running-loops` skill ship in the same patch | +| **M11** | System One / Jev typed layer | **Proposed** — optional `systemOne` config; decisions API; advisory vs block gates; not `reviewRuntime` | | **0.5.0** | Prompt TUI, Claude runtime, spend honesty, embed | **Shipped** — `agent-loop-prompt`; `runtime: claude`; list vs billed (+ cache); `docs/embed-api.md` + `SECURITY.md`; OpenRouter `:free`; Muse Spark 1.3 default | ### M6–M8 context (2026-07 competitive session) @@ -401,6 +501,7 @@ hung-worker escalate and the `check-running-loops` skill. Details in backlog **P ## Non-goals - Replacing `verify` with LLM self-assessment. +- Wiring Jev / System One as `reviewRuntime` or chat `reviewModel` (see M11). - Auto-merging or auto-closing work without human policy when HITL is configured. - Enabling full gate stack by default on trivial loops (`reviewGate` stays opt-in). - Climbing to a GitHub/Linear/Slack “software factory” product (stay the harness factories compose). diff --git a/docs/runtime-map.md b/docs/runtime-map.md index 1a66c6d..ff94127 100644 --- a/docs/runtime-map.md +++ b/docs/runtime-map.md @@ -34,6 +34,10 @@ n≥3, change one of `runtime` / `model`). Primary judge is independent: unset `reviewRuntime` → Cursor SDK. Set `reviewRuntime` + `reviewModel` to any worker runtime to keep review off Cursor quota. +**Typed review (proposed):** optional `systemOne` / Jev (`typesafe/jev-1.13`) uses the +OpenRouter **decisions** API — not a row in this runtime table and not +`reviewRuntime`. See [`system-one-review.md`](./system-one-review.md). + ## `costPreset` (detect-bound, not Auto) Named stacks so you pick **economics** instead of a model encyclopedia. Detection chooses which catalog row that means on this machine; it does not swap models mid-loop. Explicit `runtime` / `model` win. diff --git a/docs/system-one-review.md b/docs/system-one-review.md new file mode 100644 index 0000000..374f3d8 --- /dev/null +++ b/docs/system-one-review.md @@ -0,0 +1,146 @@ +--- +tags: + - documentation + - loops + - review + - agentic_ai +--- +# System One / Jev — typed review layer (proposed) + +Optional **third review layer** beside the hard verifier and the generative judge. +It does **not** replace `verify`, `reviewRuntime`, or `agent-loop-prompt` scaffold +authorship. + +**Product decision (locked):** Jev / TypeSafe System One is **not** a +`reviewRuntime` or chat judge slug. It does not emit `review.md` prose, PASS/ +ADVISORY/BLOCKERS headings, or GOAL/verify scaffold text. It returns **typed** +answers (`noul` / `choice` / `score`) plus probabilities via the OpenRouter +**System One** decisions API. + +Design context: [`loop-review-patterns.md`](./loop-review-patterns.md) (pattern 9), +build plan: [`loop-review-roadmap.md`](./loop-review-roadmap.md) (M11). + +## Stack position + +```text +verify (shell) — hard gate, unchanged + → generative primary review (reviewRuntime + reviewModel) — review.md prose + → optional reviewReproduce / reviewSecondaryRuntime — still generative + → optional systemOne — one decisions call over fenced state + → human (reviewGateHitl / HITL) — closure authority unchanged +``` + +- Runs only **after** verify is green (`verify` / `finalVerify` exit `0`). +- May run **in parallel with** or **after** generative review; harness should not + block generative review on System One when both are enabled. +- `reviewGate` behavior for prose blockers stays as shipped (impact-severity, + secondary merge, reproduce filters). System One adds a **separate** gate mode + via `systemOne.gate`. + +**Contrast with `reviewSecondaryRuntime` (M3, shipped):** secondary is still a +full agent session that **writes** `review.md` and merges gating blockers. +System One never authors markdown; it only answers a fixed question schema. + +## OpenRouter models (public) + +| Pin | OpenRouter model id | +| --- | --- | +| Pinned release | `typesafe/jev-1.13` | +| Rolling alias | `~typesafe/jev-latest` | + +Endpoint shape (TypeSafe-compatible **decisions**, not chat completions): + +```http +POST https://openrouter.ai/api/alpha/decisions +``` + +Body (conceptual): + +```json +{ + "model": "typesafe/jev-1.13", + "state": "", + "questions": [ … ] +} +``` + +Do not route Jev through OpenCode Go chat slugs or `reviewModel` strings meant +for Cursor/Cline/OpenCode SDK sessions. + +**Pricing:** OpenRouter’s public page lists on the order of ~$0.042/M input; +per-decision calls are cheap relative to a full generative judge pass. Catalog +rows in `src/loop/modelCatalog.generated.ts` / `modelPricing.generated.ts` may +lag — a follow-up sync PR is separate from this feature. + +**Auth:** reuse the same BYOK path as OpenRouter-backed workers (`OPENROUTER_API_KEY`; +see [`opencode-providers.md`](./opencode-providers.md)). Never document or paste +keys in loop bundles. + +## `loop.json` sketch (proposed) + +```json +{ + "reviewGate": true, + "reviewRuntime": "cursor", + "reviewModel": "grok-4.6-high", + "systemOne": { + "enabled": true, + "provider": "openrouter", + "model": "typesafe/jev-1.13", + "gate": "advisory" + } +} +``` + +| Field | Meaning | +| --- | --- | +| `enabled` | Master switch (default off). | +| `provider` | Start with `openrouter` only. | +| `model` | `typesafe/jev-1.13` or `~typesafe/jev-latest`. | +| `gate` | How typed answers affect the grind (below). | + +### `systemOne.gate` + +| Value | On API / parse failure | On typed “fail” signals | +| --- | --- | --- | +| `off` | N/A (disabled) | N/A | +| `advisory` | **Fail open** — log + surface in `run-report.md`; do not reopen worker | Surface scores / choices; do not reopen on noul=false alone unless harness adds explicit thresholds later | +| `block` | **Fail closed** — treat like an unrecoverable review gate error (same family as `reviewGate` stuck) | Reopen fix loop when configured thresholds trip (e.g. `does_pass` false with high confidence, or `residual_quality` below floor) | + +Threshold details belong in harness config + tests when implemented; this doc +fixes the semantics: **block** means typed layer can keep the gate open; +**advisory** never substitutes for missing generative review when `reviewGate` +expects `review.md`. + +## Question schema (harness-owned) + +Questions are **not** free-form prompts. They map from residual rubric / +`templates/REVIEWS.md` defaults so runs are comparable across loops: + +| Kind | Example id | Role | +| --- | --- | --- | +| `noul` | `does_pass` | Binary residual accept after verify | +| `score` | `residual_quality` | Calibrated 0–1 style residual strength | +| `choice` | `blocker_class` | Bucket for failure domain (`verify-bypass`, `false-closure`, …) | + +One decisions call per review cycle (after green verify), evaluating all +questions against the same fenced `state` blob (goal excerpt, diff stats, REVIEWS +overlay, optional truncated `review.md`). + +LangChain’s public Jev-as-judge experiments motivate **consistency and cost** for +repeatable rubrics; Agent Looper still treats **shell verify** as the only hard +“done” sensor and keeps generative review for prose, Guide packets, and HITL. + +## Non-goals + +- Not in `costPreset` **minmax** (minmax stays Hy3/Composer worker + Grok generative judge). +- Not `agent-loop-prompt` author for GOAL.md / `verify.sh`. +- Not a `reviewRuntime` / `reviewModel` chat slug. +- Not a replacement for impact-severity parsing or secondary generative merge. +- Not auto-enabled on trivial smokes (`reviewGate: false` loops stay unchanged). + +## Related shipped behavior + +- Primary + secondary judges: generative only — [`README.md`](../README.md) review gate flow. +- With `reviewGate: true`, secondary judge **runs** even on primary PASS (no gating bullets); with `reviewGate: false`, secondary skips on clean PASS. +- Intervention modes (Proceed / Guide / Deny / Confirm) apply to **generative** residual judgment only; System One is an additional sensor, not a fifth mode.