Four more process gaps from the same post-mortem: journey granularity, planning agents that never boot the app, a 6,153-line report generator, no cost ceiling

[MendixMau]

Companion to the gate discussion. Same post-mortem of a completed pipeline run against a large
existing Mendix application; project identity withheld. That discussion covers the one gap worth
fixing first. These are the other four, kept together because they share a cause and should
probably not be fixed one document at a time.

Each is stated as evidence plus an open question. None of them proposes a patch — in every case
the shape of the fix is the thing that needs deciding.

1. No granularity floor on what counts as an "e2e journey"

skills/testing-shape.md defines what testing a module means, but nothing rejects a
single-widget round-trip as a journey.

The flagship recorded demo of that run — the one journey chosen to show the pipeline working — was:
open a tab, set a dropdown to a role, set it back. 76 seconds of video to toggle one field twice
on one page. The other four journeys were the same shape. Five journeys, and not one of them
crossed two roles or two modules.

Nobody ever wrote down the application's actual business process. For a catalog-and-approval app
it would be a submitter proposes an item → a reviewer approves it → it becomes discoverable → a
consumer retrieves it. A journey of that shape would have exercised the real system; the five that were
built exercised widgets.

Open question: is the floor a rule in testing-shape.md ("a journey states a value-bearing
outcome crossing ≥2 roles and ≥2 modules, in one sentence, before any selector is written"), or is
it a check that refuses to register a journey whose definition names one page?

2. Planning agents with no requirement to touch the running system

54% of that run's budget — 1,211,224 tokens across 8 agent tasks — went to planning agents that
produced 3,579 lines of markdown. Not one of them booted the application, opened a page, or
attempted a login. They read source and wrote specs.

So nothing in 3,579 lines said "this app serves no stylesheet" or "these test passwords are
rewritten at container startup" — the two defects that invalidated everything built downstream. And
none asked what business process was being demonstrated, which is how #1 above happened.

skills/agent-roles.md and the runbook's stage structure actively encourage per-stage planning
agents whose deliverable is markdown.

Open question: is the answer a precondition on planning agents (a planning agent's output is
not accepted until it cites something it observed from the running system), or is it fewer planning
agents — plan inline, one page maximum, and spend the same budget on a spike that proves the thing
works?

3. The report generator is itself the defect

tests/e2e/report-render.js + report-normalize.js are 6,153 lines and produce a report whose
opening screens argue about denominators and explain why the tool refuses to print a pass rate.

The epistemics are defensible — a pass rate over a suite with invalid rungs is misleading. The
6,153 lines and the unreadable output are not. A reader cannot find what broke.

Open question: what is the fixed template? The candidate is
what ran · what passed · what broke · what to do next, one page, with a caveat that matters
written as one line rather than four paragraphs. But that's a call about what the report is for.

4. No cost ceiling on loop-until-the-instrument-agrees

The loop-until-95% pattern has no budget and no stop condition other than the instrument reading
95%. That is fine when the instrument measures something a human will read, and expensive when it
does not.

Open question: does a ceiling get agreed per experiment up front, with "stop and report" as the
terminal state instead of "keep looping"? And who sets it?

---

The caution that applies to all four

Four of these read as "the toolkit did not stop me". The reflex is to add four skills. This run
suggests the opposite problem: the toolkit already carried the right advice in ui-loop.md,
module-review.md and query-the-model.md, and it was read and not acted on.

Prefer one blocking gate over four new documents. Advice without enforcement has now failed
once at measurable cost; more advice is not obviously the fix.

Explicitly not submitted

One further item from the same post-mortem — "read the model before probing it live" (17 REST
endpoints were probed before reading the one authorization microflow that answered the question) —
is a compliance failure against skills/query-the-model.md, which already says exactly this.
It is listed here so it is visibly considered and rejected, not filed as a gap. It needs no new
rule.