From 1f674b083be4186e6fa942ddbfd3e1bc97545d7d Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 19 Sep 2026 09:26:41 +0000 Subject: [PATCH 1/7] contrib/inbox: LOOK pass / ui-loop / page-fidelity never executed on a full Stage-5 build (second occurrence) Process finding from a 3-module requirements-driven build: gate-check never re-run after scaffold, UI skills reachable only through undispatched role-agent stubs, instruments never installed, and the 2026-09-09 inbox note predicting this was still unpromoted. Five enforcement proposals, smallest first. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01XgoKZ42wYcMCBPG8Gj2PCh --- ...-19-look-pass-never-runs-benchmark-run1.md | 64 +++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 contrib/inbox/2026-09-19-look-pass-never-runs-benchmark-run1.md diff --git a/contrib/inbox/2026-09-19-look-pass-never-runs-benchmark-run1.md b/contrib/inbox/2026-09-19-look-pass-never-runs-benchmark-run1.md new file mode 100644 index 00000000..fd032a17 --- /dev/null +++ b/contrib/inbox/2026-09-19-look-pass-never-runs-benchmark-run1.md @@ -0,0 +1,64 @@ +# LOOK pass, ui-loop and page-fidelity never executed on a full Stage-5 build — second occurrence + +**From:** lowcode-vs-highcode benchmark, Mendix arm run 1 (orchestrator session) +**Date:** 2026-09-19 +**Kind:** process +**Field evidence:** A 3-module, 7-screen requirements-driven build (Mendix 11.14, mxcli 0.22.0, toolkit @ 2d9a0a2) closed with `design/ui-reviews/` empty, no `docs/PAGE-FIDELITY.tsv`, no `.claude/loop/`, `project-bin/` never populated, and `PROJECT.md` still reading "gates passed: none yet (derived by gate-check on )". The human noticed by looking at the screenshots. Same chain as `contrib/inbox/2026-09-09-ui-loop-never-runs-page-fidelity.md`, which was still unpromoted. +**Proposed target:** `skills/conversion-runbook.md` §1b (definition of done), `bin/init-project.sh` (install instruments), agent-stub dispatch / `iterative-build-loop.md`, and triage of the 2026-09-09 note together with this one. + +--- + +## What we did + +- Stages 0–4 with the toolkit (intake, BRDs, blueprint, wireframes S1–S7, ds.css, build plan). +- Stage 5 built by a lead agent that was itself a subagent, plus 14 generic worker agents in 5 + batches, each with a per-file brief naming the MDL script and the wireframe. Role-agent stubs + (`mdl-agent`, `review-agent`, …) were scaffolded but not dispatched — a subagent lead cannot + spawn subagents, and the run rules excluded them anyway. +- Every stage was declared done in the run log. `gate-check.sh` was never re-run after scaffold. +- Result: pages structurally off their wireframes (missing status chips, unformatted amounts, + absent filter chips, empty persona selector, clipped collapsed nav, raw "No items found" empty + states), on top of the scaffold `signal` theme the spec's "platform default" rule left in place. + +## Why the five UI obligations never fired (finding, reproduced by reading the artifacts) + +1. **The obligation check is only as good as the last `gate-check.sh` run.** `obligations.tsv` + row `look` would have printed PENDING for all three modules — but nothing in the build loop + requires gate-check before "done", so it printed nothing. Green-by-absence happened one level + up: absence of the *check*, not absence of the artifact. +2. **"A citation is not a read" — again.** `ui-preflight-pages.md`, `ui-loop.md`, + `module-review.md`, `page-fidelity.js` and `check-page-shell.sh` are routed through + `mdl-agent.md` / `review-agent.md` stubs and the CLAUDE.local.md situational table only. The + lead's own always-on block (mxtk wiring: RESUME.md, exec-approval, no strays) has no LOOK line, + no fidelity step, no gate-check-before-done. Workers got briefs, not skills. So the text was + in nobody's context at page-writing time. +3. **Instruments not installed.** `init-project.sh` does not copy `project-bin/page-fidelity.js` + or `check-page-shell.sh`; the skill says "run it", the project has nothing to run. +4. **Inbox not drained.** The 2026-09-09 note predicted this exactly. Ten days in the queue. + +## Should these be part of the pipeline? (position, not hypothesis) + +They are. The gap is enforcement, not content. Proposals, smallest first: + +- **A.** In the runbook's Live Checklist Protocol (§1b), make the last item of every Stage-5 + module checklist literally: `gate-check.sh 5` ran, output pasted, zero PENDING + obligations for this module. A stage reposted without it is not done. +- **B.** `init-project.sh` installs `project-bin/page-fidelity.js` and `check-page-shell.sh` + and writes the `docs/PAGE-FIDELITY.tsv` header, so the artifact check sees a stub and the + human sees zeros instead of nothing. +- **C.** Put a 6-line "before any page MDL / after any page MDL" block into the *always-on* + section of the generated `CLAUDE.local.md` (not the situational table): read + `ui-preflight-pages.md`; after exec, screenshot, run page-fidelity, append the row. This is + the inline-the-dispatch rule from CLAUDE.md authoring rule 2 applied to the lead itself. +- **D.** Hypothesis: the `look` obligation should also accept a *fidelity row ≥ threshold plus + a screenshot file* as evidence, not only `ui-review-*.html`, so single-session builds without a + review agent can still satisfy it honestly. +- **E.** Triage discipline: an inbox note older than N days that names a pipeline gap blocks the + next `init-project.sh` on that machine with a warning listing it. (Hypothesis; may be too noisy.) + +## What we are doing on the project + +Re-running the UI pass on both benchmark arms under a spec amendment: element-checklist +alignment per wireframe (present / placed / behaves), ≥95% target, header/topbar mandatory, +platform-default theme retained. Fidelity measured mechanically, LOOK pass with screenshots, +`gate-check.sh` before close. Outcome will be filed as a follow-up to this note. From d883069004864cc80709876f2d5950d2e9266ae8 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 19 Sep 2026 09:28:22 +0000 Subject: [PATCH 2/7] =?UTF-8?q?inbox=202026-09-19:=20correction=20?= =?UTF-8?q?=E2=80=94=20instruments=20were=20installed=20by=20init,=20never?= =?UTF-8?q?=20executed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01XgoKZ42wYcMCBPG8Gj2PCh --- .../2026-09-19-look-pass-never-runs-benchmark-run1.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/contrib/inbox/2026-09-19-look-pass-never-runs-benchmark-run1.md b/contrib/inbox/2026-09-19-look-pass-never-runs-benchmark-run1.md index fd032a17..d52ee14b 100644 --- a/contrib/inbox/2026-09-19-look-pass-never-runs-benchmark-run1.md +++ b/contrib/inbox/2026-09-19-look-pass-never-runs-benchmark-run1.md @@ -3,7 +3,7 @@ **From:** lowcode-vs-highcode benchmark, Mendix arm run 1 (orchestrator session) **Date:** 2026-09-19 **Kind:** process -**Field evidence:** A 3-module, 7-screen requirements-driven build (Mendix 11.14, mxcli 0.22.0, toolkit @ 2d9a0a2) closed with `design/ui-reviews/` empty, no `docs/PAGE-FIDELITY.tsv`, no `.claude/loop/`, `project-bin/` never populated, and `PROJECT.md` still reading "gates passed: none yet (derived by gate-check on )". The human noticed by looking at the screenshots. Same chain as `contrib/inbox/2026-09-09-ui-loop-never-runs-page-fidelity.md`, which was still unpromoted. +**Field evidence:** A 3-module, 7-screen requirements-driven build (Mendix 11.14, mxcli 0.22.0, toolkit @ 2d9a0a2) closed with `design/ui-reviews/` empty, no `docs/PAGE-FIDELITY.tsv`, no `.claude/loop/`, `bin/page-fidelity.js` present but never run, and `PROJECT.md` still reading "gates passed: none yet (derived by gate-check on )". The human noticed by looking at the screenshots. Same chain as `contrib/inbox/2026-09-09-ui-loop-never-runs-page-fidelity.md`, which was still unpromoted. **Proposed target:** `skills/conversion-runbook.md` §1b (definition of done), `bin/init-project.sh` (install instruments), agent-stub dispatch / `iterative-build-loop.md`, and triage of the 2026-09-09 note together with this one. --- @@ -32,8 +32,8 @@ lead's own always-on block (mxtk wiring: RESUME.md, exec-approval, no strays) has no LOOK line, no fidelity step, no gate-check-before-done. Workers got briefs, not skills. So the text was in nobody's context at page-writing time. -3. **Instruments not installed.** `init-project.sh` does not copy `project-bin/page-fidelity.js` - or `check-page-shell.sh`; the skill says "run it", the project has nothing to run. +3. **Instruments installed, never run.** `init-project.sh` copied `page-fidelity.js` + and `check-page-shell.sh` into the project `bin/`; nothing in the loop required executing them. 4. **Inbox not drained.** The 2026-09-09 note predicted this exactly. Ten days in the queue. ## Should these be part of the pipeline? (position, not hypothesis) @@ -43,8 +43,8 @@ They are. The gap is enforcement, not content. Proposals, smallest first: - **A.** In the runbook's Live Checklist Protocol (§1b), make the last item of every Stage-5 module checklist literally: `gate-check.sh 5` ran, output pasted, zero PENDING obligations for this module. A stage reposted without it is not done. -- **B.** `init-project.sh` installs `project-bin/page-fidelity.js` and `check-page-shell.sh` - and writes the `docs/PAGE-FIDELITY.tsv` header, so the artifact check sees a stub and the +- **B.** `init-project.sh` (already installs both instruments) additionally writes the `docs/PAGE-FIDELITY.tsv` header + so the artifact check sees a stub and the human sees zeros instead of nothing. - **C.** Put a 6-line "before any page MDL / after any page MDL" block into the *always-on* section of the generated `CLAUDE.local.md` (not the situational table): read From e466e4c19e95b61c40010e543e2110f8d4698617 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 19 Sep 2026 12:59:23 +0000 Subject: [PATCH 3/7] Make the gate script the last checklist item; a fidelity score is not a LOOK pass MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit conversion-runbook.md §1b rule 8: every stage and every module build ends on bin/gate-check.sh, run and pasted, with zero PENDING or each PENDING named and waived. The obligation check already reports a pass nobody performed — but only on a run, and nothing before this required one. ui-preflight-pages.md: page-fidelity.js matches identifiers in page MDL against wireframe text and cannot see placement. On the same commit, two screens scored 100% fidelity against 58% and 40% on an element checklist; a third ran the other way. Use the score for the rework curve between LOOK passes, never in place of one. init-project.sh: the generated CLAUDE.local.md now carries a before/after page-MDL item, so the pre-flight text reaches the agent that writes pages. Field evidence: lowcode-vs-highcode-benchmark results/mendix-run1 (closed blind), results/mendix-run1b (seven screens at 0/18/40/40/58/58/69% baseline), and results/highcode-run1b as the non-Mendix control (43% and 56% baseline on two screens) — a pipeline gap, not a platform one. Withdraws proposal D of the 2026-09-19 inbox note as field-refuted. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01XgoKZ42wYcMCBPG8Gj2PCh --- CHANGELOG.md | 21 +++++++++++++++++++++ bin/init-project.sh | 9 +++++++++ skills/conversion-runbook.md | 17 +++++++++++++++++ skills/ui-preflight-pages.md | 19 +++++++++++++++++++ 4 files changed, 66 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2cb3de4f..117a3e81 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,27 @@ moment updating it became a separate chore). One line per change: Kinds: `new` · `fix` · `learn` (a skill/learning) · `process` (rules, templates, CI). Credit the person or project that surfaced the change — the credit line is the thank-you. +## 2026-09-19 +- process(runbook, ui-preflight, init-project): **a full Stage-5 build closed with seven screens, + a green mxbuild gate, and a `look` obligation nobody had ever discharged — because nothing in + the Live Checklist Protocol required the gate script to run.** `gate-check.sh` had executed once, + at scaffold time; the obligation check reports a pass nobody performed, but only on a run. + Three changes, all from the same field runs: `conversion-runbook.md` §1b gains **rule 8** — the + last checklist item of every stage and every module build is `bin/gate-check.sh` run and pasted, + zero `PENDING` or each one named and waived; `ui-preflight-pages.md` gains **"a fidelity score is + not a LOOK pass"**, because `page-fidelity.js` matches identifiers in page MDL against wireframe + text and cannot see placement — measured on the same commit, two screens scored 100% fidelity + while an element checklist put them at 58% and 40%, and a third ran the other way (55% fidelity, + 69% checklist); `init-project.sh` writes a **before/after page-MDL item** into the generated + `CLAUDE.local.md`, so the pre-flight text reaches the agent that actually writes pages (a citation + is not a read). Field evidence: `lowcode-vs-highcode-benchmark` runs `results/mendix-run1/` + (the build that closed blind) and `results/mendix-run1b/` (the deliberate LOOK pass that scored + the seven screens at 0/18/40/40/58/58/69% baseline, five iterations and 75 screenshots later at + 90-100%), plus `results/highcode-run1b/` as the non-Mendix control — the high-code arm's baseline + was 43% and 56% on two screens, so this is a pipeline gap, not a platform one. Withdraws + proposal D of `contrib/inbox/2026-09-19-look-pass-never-runs-benchmark-run1.md` (accept a fidelity + row as LOOK evidence) as field-refuted. — MendixMau, PRD benchmark + ## 2026-09-17 - process(register): **the toolkit never mentioned `mxcli brain`, while mxcli writes "read `docs/brain/project.md` first" into every project's CLAUDE.md** — so a wired project ran two diff --git a/bin/init-project.sh b/bin/init-project.sh index 2e62dbba..67c5ddb3 100755 --- a/bin/init-project.sh +++ b/bin/init-project.sh @@ -327,6 +327,15 @@ This project uses the shared toolkit at \`$TOOLKIT_ROOT\`. For ANY pipeline work 3. **Every gate question is asked in chat, then the turn ends and waits** (runbook §1). \`ASSUMED\` is earned by asking — the user said "you decide" — never by skipping. 4. **Before calling a stage done:** \`$TOOLKIT_ROOT/bin/gate-check.sh \`. +5. **Before any page MDL:** read \`$TOOLKIT_ROOT/skills/ui-preflight-pages.md\` in full — + its *text*, not a reference to it. If page drafting is delegated, the dispatch prompt + carries the text or the subagent reads the file as its first action. + **After any page MDL:** \`bash bin/check-page-shell.sh\`, then + \`node $TOOLKIT_ROOT/project-bin/page-fidelity.js