Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 37 additions & 4 deletions docs/loop-review-patterns.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,12 @@ irrelevant blockers or declare itself done on vibes.
| Reviewer | LLM judgment | scoped to the structural/architectural residual specs can't capture. |
| Human | closure authority | HITL sign-off for subjective or high-stakes decisions. |

Optional **typed decision layer** (System One / Jev — proposed): sits beside the
generative reviewer, never replaces it. After green verify, a cheap decisions API
answers fixed-schema questions (`noul` / `choice` / `score`) over fenced state;
it does not write `review.md` or scaffold GOAL/verify. See pattern 9 and
[`system-one-review.md`](./system-one-review.md).

Key warning (Specification as Quality Gate, 2026): an LLM reviewing
LLM-generated code is **circular** without an external reference — both share a
training distribution and echo each other's errors. So `verify` is the real gate;
Expand Down Expand Up @@ -109,6 +115,34 @@ separate cross-loop reviewer design, validated. Their taxonomy also reminds us:
**start simple, use patterns selectively** — `reviewGate` off is the right path for
trivial tasks; the full gating stack is for high-stakes work.

### 9. Typed decision layer (System One / Jev) beside generative review
LangChain and TypeSafe have experimented with **Jev** as a judge: same rubric,
typed outputs, and probability scores instead of free-form critique — useful for
**consistency and cost** on repeatable residual checks. Agent Looper maps that to
an optional layer **after verify**, not a new `reviewRuntime`:

1. **Verifier** — unchanged hard gate.
2. **Generative reviewer** — `reviewRuntime` / `reviewModel` (and optional
`reviewSecondaryRuntime`) still produce `review.md`, Guide packets, and
impact-severity blockers; `agent-loop-prompt` still uses this judge to *write*
GOAL + verify.
3. **System One** — one OpenRouter decisions call (`typesafe/jev-1.13` or
`~typesafe/jev-latest`) with harness-fixed questions derived from REVIEWS
residual defaults (e.g. noul `does_pass`, score `residual_quality`, choice
`blocker_class`).

Jev does **not** generate text. Wiring it as a chat judge would break scaffold
and `reviewGate` prose contracts. Instead, `systemOne.gate` controls grind impact:
`advisory` surfaces typed answers in `run-report.md` (fail open on API errors);
`block` may reopen the loop on explicit thresholds (fail closed on API/parse
errors). Typed layer may run parallel to or after generative review; it never
replaces verify or human closure (`reviewGateHitl`).

Maps to us: complements M3 multi-family **generative** secondary judge — secondary
still merges markdown blockers; System One adds a calibrated sensor for ops
metrics and optional cheap gating. Not on `costPreset` minmax. Detail:
[`system-one-review.md`](./system-one-review.md); roadmap M11.

## The meta-loop (your "10 loops' diffs → review")

Scaling to a software factory is **not** bolting more gating onto every loop. It is
Expand All @@ -132,10 +166,9 @@ blueprint for building it.
the right line, (c) did it hallucinate a blocker. That's how the 75% false-blocker
inflation was discovered — and the only way to know if `reviewGate` helps or
thrashes.
- **Highest-ROI next step:** add an **impact-severity contract** to blockers in
`reviewVerdict` (gate only on `error`-with-impact; downgrade the rest to advisory).
It is the single change the 2026 data says most directly stops the irrelevant-
blocker fix-loop. Full build order, acceptance criteria, and Taskwarrior mapping:
- **Highest-ROI next step (shipped M1):** impact-severity contract in
`reviewVerdict`. Next optional sensor: typed System One layer (roadmap M11) —
not a substitute for generative `reviewGate`. Full build order:
[`loop-review-roadmap.md`](./loop-review-roadmap.md).
- **Encode misses as system improvements:** when a loop result doesn't meet
standard, turn the fix into a reusable skill/check (pattern 6) or a
Expand Down
103 changes: 102 additions & 1 deletion docs/loop-review-roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,13 @@ Concrete implementation plan for the upgrades called out in
[`loop-review-patterns.md`](./loop-review-patterns.md). Principles stay fixed:

1. **Verifier** is the hard gate (`verify` / `finalVerify`).
2. **Reviewer** is gated judgment on top (never replaces verify).
2. **Reviewer** is gated judgment on top (never replaces verify) — generative
`reviewRuntime` / `reviewModel` (+ optional `reviewSecondaryRuntime`).
3. **Human** remains closure authority for residual / high-stakes calls (`reviewGateHitl`).

Optional **typed decision layer** (`systemOne` / Jev) is documented as M11; it is
not a fourth principle and not a `reviewRuntime` substitute.

Research context and citations live in the patterns doc. This file is the
**build order, acceptance criteria, and file touch list**.

Expand Down Expand Up @@ -52,6 +56,7 @@ Open Taskwarrior project for **Agent Looper** (slug **`agent-loop`**). Use **UUI
| M3 | `adf66bf8-d52a-43e2-8009-756649cc32b2` | Multi-family secondary judge (shipped) |
| M4 | `fe3f4076-b997-4d28-a59a-baf720c28e5d` | Verification-as-skill |
| M5 | `06dec3c5-b35d-4e8a-bb95-c0f2a9ae4f00` | Cross-loop meta-review CLI |
| M11 (proposed) | *(no UUID yet)* | System One / Jev typed review layer |

Other Agent Looper backlog: `17bfc1cd-bf5d-43a7-9b8b-9bf7658aaa07` (extract review-gate) — **done**,
`de4144f2-9e6a-4cf6-8943-81efc49d4c5c` (loopRisk profiles) — **shipped**,
Expand All @@ -69,6 +74,7 @@ Other Agent Looper backlog: `17bfc1cd-bf5d-43a7-9b8b-9bf7658aaa07` (extract revi
| 3 | Multi-provider / multi-family review | Medium–high (bias reduction) | L | #1 useful first |
| 4 | Verification-as-skill | High for quality, orthogonal | M | — |
| 5 | Cross-loop meta-reviewer | Factory scale | L | #1–2 data shape helps |
| 6 | System One / Jev typed layer | Medium (cost + consistency sensor) | M | Orthogonal to #1–3; needs generative review contract stable |

Pilot rule (from patterns): tune `maxReviewCycles` / `unparseableReviewRetries` on a
handful of real loops before locking defaults for #1–3.
Expand Down Expand Up @@ -342,6 +348,99 @@ much better; not a hard code dependency.

---

## 6. System One / Jev typed review layer

### Status

**Proposed** — design + docs only; no harness implementation on the M11 doc PR.
Taskwarrior UUID pending.

### Problem

Generative judges are the right tool for `review.md`, Guide packets, and
`agent-loop-prompt` scaffold prose, but they are expensive and noisy for
**repeatable residual rubrics** (pass/fail, quality score, blocker class). Typed
decision models (TypeSafe Jev via OpenRouter System One) answer fixed questions
with probabilities — useful as a **sensor**, not as a replacement for verify or
markdown review.

### Design

**Not** `reviewRuntime`. Separate `loop.json` block:

```json
{
"reviewGate": true,
"reviewRuntime": "cursor",
"reviewModel": "grok-4.6-high",
"systemOne": {
"enabled": true,
"provider": "openrouter",
"model": "typesafe/jev-1.13",
"gate": "advisory"
}
}
```

| `systemOne.gate` | Behavior |
| --- | --- |
| `off` | Disabled (default when block omitted). |
| `advisory` | Log + `run-report.md`; API/parse errors **fail open**; do not reopen worker on typed alone unless product adds explicit thresholds. |
| `block` | Configured thresholds may reopen fix loop; API/parse errors **fail closed** (same seriousness as `reviewGate` infrastructure failures). |

**API:** OpenRouter `POST …/api/alpha/decisions` with `model`, `state`, `questions`
— not chat completions. Models: `typesafe/jev-1.13`, alias `~typesafe/jev-latest`.

**State:** fenced bundle (GOAL excerpt, diff summary, `REVIEWS.md`, optional
truncated `review.md`).

**Questions:** harness-fixed schema from residual defaults — examples: noul
`does_pass`, score `residual_quality`, choice `blocker_class`. One call per
review cycle after green verify.

**Ordering:** after verify green; may run parallel to or after generative primary/
secondary review. Shipped `reviewSecondaryRuntime` stays generative (markdown
merge); System One does not author `review.md`.

**Auth:** existing `OPENROUTER_API_KEY` BYOK path ([`opencode-providers.md`](./opencode-providers.md)).

Full API notes: [`system-one-review.md`](./system-one-review.md). Pattern motivation:
[`loop-review-patterns.md`](./loop-review-patterns.md) §9.

### Implementation sketch

| Area | Work |
| --- | --- |
| `loopConfig.ts` | `systemOne` zod object; default disabled |
| `loopPostReview.ts` | Invoke decisions client after verify; respect `gate` |
| New `src/review/systemOne*.ts` | OpenRouter decisions client, question templates, threshold eval |
| `run-report.md` | Advisory fields + block reasons |
| `failure-domains.ndjson` | Optional `system_one_*` reasons |
| Catalog follow-up | Separate PR to `pnpm sync:models` when models.dev lists Jev |

### Acceptance criteria

- [ ] With `systemOne.enabled: false` (or unset), behavior identical to today.
- [ ] Generative `reviewGate` + secondary merge unchanged when System One off.
- [ ] `gate: advisory` never blocks completion on API/parse failure.
- [ ] `gate: block` fails closed on API/parse failure.
- [ ] No code path treats Jev as `reviewRuntime` or `agent-loop-prompt` judge.
- [ ] Docs + dogfood loop example; model ids exactly `typesafe/jev-1.13` /
`~typesafe/jev-latest`.

### Non-goals

- `costPreset` minmax inclusion.
- Scaffold or `review.md` authorship.
- Chat-slug `reviewModel` for Jev.
- Replacing impact-severity or reproduce-before-report.

### Depends on

Soft: stable generative review pipeline (#1–3 shipped). Hard: none.

---

## Validation experiment (shared)

Before declaring #1–3 “done”, run a small offline experiment (patterns doc):
Expand Down Expand Up @@ -370,6 +469,7 @@ PR that lands #1.
| **M8** | Batch rubrics / Auto model (optional) | **Shipped** — batch `{path,rubric}`; computer-use templates; Auto **blocked** pending SDK (`docs/cursor-auto-router.md`) |
| **M9** | Linear Loops governance steals | **Shipped (docs/templates)** — prove→freeze, draft discipline, run-report as audit surface, `LOOP.permissions.example.md`, tool default-deny. No harness publish-snapshot yet |
| **M10** | Lunar graph scoreboard | **0.4.5** — docs (edge sentences + GOAL/preflight spec fields) + `run-report.md` report card (phase time, kill rate, retries, HITL, writer vs referee $). Hung-worker escalate + `check-running-loops` skill ship in the same patch |
| **M11** | System One / Jev typed layer | **Proposed** — optional `systemOne` config; decisions API; advisory vs block gates; not `reviewRuntime` |
| **0.5.0** | Prompt TUI, Claude runtime, spend honesty, embed | **Shipped** — `agent-loop-prompt`; `runtime: claude`; list vs billed (+ cache); `docs/embed-api.md` + `SECURITY.md`; OpenRouter `:free`; Muse Spark 1.3 default |

### M6–M8 context (2026-07 competitive session)
Expand Down Expand Up @@ -401,6 +501,7 @@ hung-worker escalate and the `check-running-loops` skill. Details in backlog **P
## Non-goals

- Replacing `verify` with LLM self-assessment.
- Wiring Jev / System One as `reviewRuntime` or chat `reviewModel` (see M11).
- Auto-merging or auto-closing work without human policy when HITL is configured.
- Enabling full gate stack by default on trivial loops (`reviewGate` stays opt-in).
- Climbing to a GitHub/Linear/Slack “software factory” product (stay the harness factories compose).
Expand Down
4 changes: 4 additions & 0 deletions docs/runtime-map.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@ n≥3, change one of `runtime` / `model`).

Primary judge is independent: unset `reviewRuntime` → Cursor SDK. Set `reviewRuntime` + `reviewModel` to any worker runtime to keep review off Cursor quota.

**Typed review (proposed):** optional `systemOne` / Jev (`typesafe/jev-1.13`) uses the
OpenRouter **decisions** API — not a row in this runtime table and not
`reviewRuntime`. See [`system-one-review.md`](./system-one-review.md).

## `costPreset` (detect-bound, not Auto)

Named stacks so you pick **economics** instead of a model encyclopedia. Detection chooses which catalog row that means on this machine; it does not swap models mid-loop. Explicit `runtime` / `model` win.
Expand Down
146 changes: 146 additions & 0 deletions docs/system-one-review.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
---
tags:
- documentation
- loops
- review
- agentic_ai
---
# System One / Jev — typed review layer (proposed)

Optional **third review layer** beside the hard verifier and the generative judge.
It does **not** replace `verify`, `reviewRuntime`, or `agent-loop-prompt` scaffold
authorship.

**Product decision (locked):** Jev / TypeSafe System One is **not** a
`reviewRuntime` or chat judge slug. It does not emit `review.md` prose, PASS/
ADVISORY/BLOCKERS headings, or GOAL/verify scaffold text. It returns **typed**
answers (`noul` / `choice` / `score`) plus probabilities via the OpenRouter
**System One** decisions API.

Design context: [`loop-review-patterns.md`](./loop-review-patterns.md) (pattern 9),
build plan: [`loop-review-roadmap.md`](./loop-review-roadmap.md) (M11).

## Stack position

```text
verify (shell) — hard gate, unchanged
→ generative primary review (reviewRuntime + reviewModel) — review.md prose
→ optional reviewReproduce / reviewSecondaryRuntime — still generative
→ optional systemOne — one decisions call over fenced state
→ human (reviewGateHitl / HITL) — closure authority unchanged
```

- Runs only **after** verify is green (`verify` / `finalVerify` exit `0`).
- May run **in parallel with** or **after** generative review; harness should not
block generative review on System One when both are enabled.
- `reviewGate` behavior for prose blockers stays as shipped (impact-severity,
secondary merge, reproduce filters). System One adds a **separate** gate mode
via `systemOne.gate`.

**Contrast with `reviewSecondaryRuntime` (M3, shipped):** secondary is still a
full agent session that **writes** `review.md` and merges gating blockers.
System One never authors markdown; it only answers a fixed question schema.

## OpenRouter models (public)

| Pin | OpenRouter model id |
| --- | --- |
| Pinned release | `typesafe/jev-1.13` |
| Rolling alias | `~typesafe/jev-latest` |

Endpoint shape (TypeSafe-compatible **decisions**, not chat completions):

```http
POST https://openrouter.ai/api/alpha/decisions
```

Body (conceptual):

```json
{
"model": "typesafe/jev-1.13",
"state": "<fenced bundle: GOAL excerpt, diff summary, REVIEWS.md, optional review.md>",
"questions": [ … ]
}
```

Do not route Jev through OpenCode Go chat slugs or `reviewModel` strings meant
for Cursor/Cline/OpenCode SDK sessions.

**Pricing:** OpenRouter’s public page lists on the order of ~$0.042/M input;
per-decision calls are cheap relative to a full generative judge pass. Catalog
rows in `src/loop/modelCatalog.generated.ts` / `modelPricing.generated.ts` may
lag — a follow-up sync PR is separate from this feature.

**Auth:** reuse the same BYOK path as OpenRouter-backed workers (`OPENROUTER_API_KEY`;
see [`opencode-providers.md`](./opencode-providers.md)). Never document or paste
keys in loop bundles.

## `loop.json` sketch (proposed)

```json
{
"reviewGate": true,
"reviewRuntime": "cursor",
"reviewModel": "grok-4.6-high",
"systemOne": {
"enabled": true,
"provider": "openrouter",
"model": "typesafe/jev-1.13",
"gate": "advisory"
}
}
```

| Field | Meaning |
| --- | --- |
| `enabled` | Master switch (default off). |
| `provider` | Start with `openrouter` only. |
| `model` | `typesafe/jev-1.13` or `~typesafe/jev-latest`. |
| `gate` | How typed answers affect the grind (below). |

### `systemOne.gate`

| Value | On API / parse failure | On typed “fail” signals |
| --- | --- | --- |
| `off` | N/A (disabled) | N/A |
| `advisory` | **Fail open** — log + surface in `run-report.md`; do not reopen worker | Surface scores / choices; do not reopen on noul=false alone unless harness adds explicit thresholds later |
| `block` | **Fail closed** — treat like an unrecoverable review gate error (same family as `reviewGate` stuck) | Reopen fix loop when configured thresholds trip (e.g. `does_pass` false with high confidence, or `residual_quality` below floor) |

Threshold details belong in harness config + tests when implemented; this doc
fixes the semantics: **block** means typed layer can keep the gate open;
**advisory** never substitutes for missing generative review when `reviewGate`
expects `review.md`.

## Question schema (harness-owned)

Questions are **not** free-form prompts. They map from residual rubric /
`templates/REVIEWS.md` defaults so runs are comparable across loops:

| Kind | Example id | Role |
| --- | --- | --- |
| `noul` | `does_pass` | Binary residual accept after verify |
| `score` | `residual_quality` | Calibrated 0–1 style residual strength |
| `choice` | `blocker_class` | Bucket for failure domain (`verify-bypass`, `false-closure`, …) |

One decisions call per review cycle (after green verify), evaluating all
questions against the same fenced `state` blob (goal excerpt, diff stats, REVIEWS
overlay, optional truncated `review.md`).

LangChain’s public Jev-as-judge experiments motivate **consistency and cost** for
repeatable rubrics; Agent Looper still treats **shell verify** as the only hard
“done” sensor and keeps generative review for prose, Guide packets, and HITL.

## Non-goals

- Not in `costPreset` **minmax** (minmax stays Hy3/Composer worker + Grok generative judge).
- Not `agent-loop-prompt` author for GOAL.md / `verify.sh`.
- Not a `reviewRuntime` / `reviewModel` chat slug.
- Not a replacement for impact-severity parsing or secondary generative merge.
- Not auto-enabled on trivial smokes (`reviewGate: false` loops stay unchanged).

## Related shipped behavior

- Primary + secondary judges: generative only — [`README.md`](../README.md) review gate flow.
- With `reviewGate: true`, secondary judge **runs** even on primary PASS (no gating bullets); with `reviewGate: false`, secondary skips on clean PASS.
- Intervention modes (Proceed / Guide / Deny / Confirm) apply to **generative** residual judgment only; System One is an additional sensor, not a fifth mode.
Loading