Gate needed between "plan written" and "build the harness": three skills already say to look at the running app, none of them blocks

[MendixMau]

Title: The toolkit has no blocking gate between "plan is written" and "build a full e2e
harness", so a project can spend most of its budget building and running instruments against an
application that never rendered correctly and whose test credentials didn't hold — and every
existing document that says to check this is advisory, not enforced

Summary

A pipeline run spent ~2.83M tokens across 23 agent tasks. 54% of that (1,211,224 tokens, 8 agent
tasks) went to planning agents that produced 3,579 lines of markdown and never booted the
application. The run then built a 15,018-line Playwright/OTel/OQL test harness and five "e2e
journeys" and ran them to green against an application that was serving no stylesheet at all
(every page rendered as raw unstyled HTML) and whose test-user passwords were being rewritten on
every container startup (so logins that worked at one point silently stopped working later,
undetected). Both defects were real, application-breaking, and were found later, by a human
running things and looking, at full sunk cost.

The toolkit is not silent on this. skills/ui-loop.md, skills/module-review.md stage 4 (the
LOOK), and project-bin/check-design-reaches-app.sh all already say, in different words, "look at
the running app before you trust an instrument." None of the three is wired to anything that
blocks work from proceeding. They are read, not enforced, and on this run they were not acted
on. The candidate fix is therefore one blocking gate, not a fourth document repeating the point —
but that is exactly the design decision that wants review, not a decision this report makes for
the maintainer.

Alongside the missing gate, the same post-mortem found: no floor on what counts as an e2e
"journey" (a single-widget round-trip was accepted as one); a planning-agent pattern with no
requirement that a planning agent touch the running system; a report generator whose output is too
long to read; and no cost ceiling on the loop-until-a-target-score pattern. One further item in the
same post-mortem — "read the model before probing it live" — is judged a compliance failure
against an existing skill (skills/query-the-model.md), not a gap, and needs no new rule.

Environment

• Toolkit: the skill set and bin//project-bin/ scripts under
  mxcli-project-toolkit-app-analysis, as pulled into a consuming Mendix/mxcli project on
  2026-09-17.
• Consuming project: a large existing Mendix application undergoing the toolkit's standard
  discover → design → build → test pipeline (Stages P–7 of skills/conversion-runbook.md).
  Project identity, module names, and any credentials are withheld here — see Part 2.
• Files inspected directly for this report: skills/ui-loop.md, skills/module-review.md,
  skills/query-the-model.md, skills/testing-shape.md, skills/agent-roles.md,
  skills/conversion-runbook.md, project-bin/check-design-reaches-app.sh, bin/gate-check.sh.

Steps to reproduce

This is a process gap, not a crashing command, so "reproduce" means: follow the runbook on a
project designed to expose the gap, and confirm nothing stops you.

1. Stand up (or use) a project where the running application is misconfigured in a way that is
   invisible from the model or from source — e.g. the compiled theme/stylesheet never actually
   reaches the served app (a known, previously-documented failure shape; see
   skills/learned-css-that-never-applied.md), or a startup script rewrites demo-user passwords
   so a login that worked once stops working intermittently.
2. Follow skills/conversion-runbook.md Stages P through 4 as written: intake, triage, BRDs,
   architecture, build plan. Confirm that none of these stages requires booting the app or opening
   a browser — they are read/write-a-document stages by design, and skills/agent-roles.md
   describes exactly this "discover/design" split with no running-system requirement attached to
   the planning agents (architect-agent's own tool rights explicitly exclude mxcli).
3. Enter Stage 5 (Build). Note that skills/conversion-runbook.md §Stage 5 does say the stage
   opens with a walking skeleton that must be "proven in the running app" with one screenshot —
   this is the one existing checkpoint closest to a real gate. Confirm whether it was invoked, and
   if it was, confirm what it actually checked: one screenshot of one page early in the build is
   not a re-check after later pages, config changes, or container restarts change what is served.
4. Continue building pages/microflows across multiple modules. After each module, run
   skills/module-review.md's stage 4 (LOOK) and project-bin/check-design-reaches-app.sh and
   confirm they exist and say the right thing — then confirm that skipping them produces no
   error, no non-zero exit, and no entry in bin/gate-check.sh's stage report. Run
   bin/gate-check.sh <project-root> at any point in Stage 5 or 6 and read its stage table: it has
   no row that asks "has a human looked at a rendered page in the last N build scripts" or "did the
   walking-skeleton screenshot get re-taken since."
5. Proceed straight to Stage 6 (Test) and let test-agent build the full Playwright/OTel/OQL
   harness described in skills/testing-shape.md and skills/module-review.md stage 3 (PROVE).
   Observe that none of the mechanical rungs (Playwright, OQL, monkey) are gated on the LOOK stage
   having actually happened — verify-module.sh runs the mechanical instruments regardless.
6. Write one "journey" that is a single widget round-trip (e.g. open a page, flip one dropdown to
   a different value, flip it back, close the page) and confirm nothing in skills/testing-shape.md
   or the journey-definition convention rejects it as too small to be a "journey" — there is no
   granularity floor stated anywhere in the shape.
7. Expected result of this checklist: at every one of these points, work proceeds. The instrument
   that would have caught the misconfiguration (a person looking at a screenshot) is optional at
   every stage where it appears, and is never the thing that has to turn green before the next,
   more expensive stage is allowed to start.

Expected

Before any test, scan, or scorer is written, something in the pipeline blocks further work
until: the app is confirmed to boot, at least a few real pages are confirmed to render with the
intended stylesheet, and login as each defined persona is confirmed to hold. If any of those fail,
that failure is the only work item — nothing after Stage 5's opening screenshot proceeds until it
is fixed. This is stated as a design intent in three places already (ui-loop.md,
module-review.md stage 4, check-design-reaches-app.sh); none of the three is wired to a gate
that can fail a stage in bin/gate-check.sh or otherwise stop the pipeline from advancing.

Actual

• ui-loop.md is explicitly a cadence recommendation ("run it ten times per module"), not a
  gate — it says outright that it does not discharge Gate: UI.
• module-review.md stage 4 (LOOK) is real and well-specified, but it fires at module close,
  which on the run in question was long after the harness and journeys already existed and had
  already reported green.
• check-design-reaches-app.sh measures whether tokens/classes from a ported design system
  reached the built app — a real, useful check — but nothing requires it to run, and nothing fails
  a stage if it is skipped or if it is run and fails.
• bin/gate-check.sh evaluates Stages 0–7 mechanically (file existence, grep-based checks) and
  reports PASS/FAIL/MANUAL/PENDING per stage, but has no check in its stage table for "has a human
  confirmed the running app renders and authenticates," at Stage 5 or Stage 6.
• Consequently: 1,211,224 tokens (54% of the run) were spent by 8 planning agents that never
  booted the app, and a 15,018-line test harness plus five recorded "e2e journeys" were built and
  run to green against an application serving no stylesheet, with passwords that did not hold
  across container restarts. Both defects were found only afterward, by a human running the app
  directly.

Root cause

Architectural, not a code defect: advice without a blocking gate did not survive contact. Three
toolkit artifacts already tell an agent to look at the running app before trusting an instrument.
None of the three is a stage-gate check inside bin/gate-check.sh, none of the three has a
required, re-runnable trigger point before the expensive stage (harness construction) is allowed to
start, and none of the three is scoped to catch a regression (an app that rendered fine at the
Stage-5 walking-skeleton screenshot and then stopped rendering fine later, e.g. because a container
restart between Stage 5 and Stage 6 rewrote the demo passwords). The runbook's own stage split
(agent-roles.md: discover/design agents vs. build/test agents) structurally allows — and on this
run rewarded — a large fraction of the budget being spent by agents that have no tool rights to run
the app at all.

Two secondary, independently real gaps compound this: (1) testing-shape.md and the journey
convention set no granularity floor, so a single-widget toggle is accepted as a "journey" with the
same standing as a genuine cross-role business process; (2) the report generators
(report-render.js / report-normalize.js, toolkit-lineage) are long enough (6,153 lines produced
in this run) that a reader cannot find "what broke" in the output even after the harness ran.

Severity

High — process-level, budget-destroying, and silent. It does not throw an error or a non-zero exit
anywhere; it produces a fully green report over a broken application. The toolkit's own prior
memory (look-at-the-app-not-just-the-instrument, suspect-the-instrument-before-the-app) shows
this is not a first occurrence of the shape.

Trigger boundary

Gate present today	Fires when	Blocks anything downstream?
Stage 5 walking-skeleton screenshot	Once, at Stage 5 open	No re-check; a later regression (config drift, container restart) is invisible to it
ui-loop.md	After every page-building script, by convention	No — advisory cadence, explicitly does not discharge Gate: UI
module-review.md stage 4 (LOOK)	At module close	No hard stop before it — a harness can be built and run to green before any module is ever closed
check-design-reaches-app.sh	Whenever someone chooses to run it	No — not wired into bin/gate-check.sh's stage table
bin/gate-check.sh stage table	Any time, on demand	Reports PASS/FAIL per stage from file-existence/grep checks; has no "human looked at a rendered page" row

Where each rule belongs — toolkit gap, compliance failure, or project-local

#	Rule (post-mortem's framing)	Verdict	What in the toolkit already exists, and what's missing
1	Look before you instrument	Toolkit gap — a missing GATE, not a missing rule	ui-loop.md, module-review.md stage 4, and check-design-reaches-app.sh already say this. Nothing blocks instrument-building until the app is verified renderable and logins hold.
2	A journey is a business process, not a widget interaction	Toolkit gap — genuinely absent	testing-shape.md defines what testing a module means but sets no granularity floor for a journey. Nothing rejects a single-widget round-trip as an "e2e journey."
3	No (unaccountable) planning agents	Toolkit gap — structural	agent-roles.md and the runbook's stage structure encourage per-stage planning agents whose sole deliverable is markdown. No rule requires a planning agent to have touched the running system.
4	One-page reports	Toolkit gap — the report generator is the defect	report-render.js / report-normalize.js are toolkit-lineage. The underlying epistemics (state your denominator, don't print a bare pass rate) are defensible; the length and unreadability of the output are not.
5	Nine-section case-study rule only for topics that shipped something	Project-local — NOT a toolkit issue	Comes from the consuming project's own CLAUDE.md "Case study rule." Fix it there, not here.
6	Budget per experiment, enforced	Toolkit gap	The loop-until-a-target-score pattern has no cost ceiling and no stop condition other than the instrument reaching its target.
7	Read the model before probing it live	Compliance failure — rule already exists	query-the-model.md says exactly this: query the model, then read the source, then ask the human. It was not followed on this run. No new rule needed.

Five toolkit gaps (1, 2, 3, 4, 6), one compliance failure against an existing skill (7), one
project-local item (5). Of the five gaps, Rule 1 is the highest-value fix by a wide margin — it
is the only one of the five that would have prevented the expensive failure on this run, and per
the framing above it needs a gate, not another skill.

A concrete illustration of Rule 2, from the same run: the flagship recorded demo journey was open
a tab → set a dropdown to a different value → set it back. 76 seconds of recorded video toggling
one field twice on one page. The other four "journeys" built in the same run followed the same
shape. No document in the toolkit's testing-shape convention gave anyone a reason to reject that as
too small to count.

How to improve — ranked

1. One blocking gate for Rule 1 (highest priority; the only fix that would have prevented this
   run's cost). Add a stage-gate check to bin/gate-check.sh — a Stage 5/6 row, not a new
   skill — that fails the stage unless there is a recent, dated record of: the app booted, N
   sampled pages rendered with the intended stylesheet (screenshot or check-design-reaches-app.sh
   evidence), and each defined persona's login was exercised and held. This turns ui-loop.md's
   and module-review.md stage 4's existing advice into something gate-check.sh can actually
   fail on, rather than a convention an agent can silently not act on. Needs a decision on: how
   "recent" (re-check after every N build scripts? after any container restart?) and what counts
   as sufficient evidence — a question for review, not one this report answers.
2. A granularity floor for Rule 2, added to testing-shape.md (or the journey-definition
   convention it points at): a journey must name a value-bearing outcome, crossing at least two
   roles or two modules, in one sentence, before any selector is written. Rejects the
   single-widget-toggle shape by definition rather than by review.
3. A running-system touchpoint for Rule 3, added to agent-roles.md's planning-agent
   description (or the runbook's Stage 3/4 entries) — a planning agent's deliverable is not
   accepted as closing its stage unless it includes evidence the running app was opened at least
   once during that stage's work.
4. A length/format constraint for Rule 4, applied to report-render.js/report-normalize.js —
   keep the denominator-stating epistemics, but cap the reader-facing surface to "what ran · what
   passed · what broke · what to do next," pushing the methodology discussion to an appendix the
   reader can skip.
5. A cost ceiling for Rule 6, added wherever the loop-until-target-score pattern is specified
   (the 95%-instrument convention) — an explicit token or iteration ceiling agreed before the loop
   starts, with "stop and report" as a valid, non-failing outcome at the ceiling.
6. Rule 7 needs no new rule. query-the-model.md already states the correct order (query the
   model, then read the source, then ask the human). This is a compliance failure to flag in
   review/retro process, not a toolkit change.
7. Rule 5 is out of scope for this repo. It belongs to the consuming project's own CLAUDE.md
   and should be raised there, not here.

Trigger boundary addendum — why this is filed as "needs review," not "here is the patch"

The post-mortem this report is based on explicitly cautions against the reflex of adding more
skills: four of the five gaps above are shaped like "the toolkit did not stop me," and the toolkit
already carried correct advice for the costliest one (Rule 1) in three separate files that were
read and not acted on. Writing a fourth document restating the same point is not expected to change
the outcome. The recommendation above is deliberately one blocking gate change (item 1) plus three
smaller convention edits (items 2–5) — but which check gates which stage, how strict it should be,
and whether it risks false-blocking a project that has a legitimate reason to defer a full design
port, are maintainer decisions. Nothing here should be applied without that review.

---