[roodboi]

HACK-1169 (under WU10 HACK-1123). This freezes revision 1 of the v5 matched application benchmark protocol before any trials, and records the known storage-witness startup cost that the user accepted. No benchmark was run. The PR claims no performance or resource result.

Head 89421c82 reworks the first commit (15496fa0) after independent verification. That verification found that not_applicable data could be accepted, that several weakenings of the frozen protocol still validated, that malformed input caused tracebacks, and that some Gate C coverage was missing. Details are in "Rework" below.

Final behavior

• docs/reference/application-benchmark-protocol.md and its machine-readable copy, scripts/application-benchmark-protocol.json, freeze:
  • Artifacts. Candidate v5.0.0-next.5 at 28de2f89 and stable v4.1.1, v4.2.0 and v4.2.1, each with its tag commit, darwin-arm64 asset name and release asset SHA-256 (read from the GitHub release API).
  • Host. M3 Max with 16 CPUs and 128 GiB, macOS 27.0, Docker CLI 29.4.0, Compose v5.1.2, and OrbStack 2.2.3 (build 20963) at 16 CPUs / 71680 MiB. Any drift refuses the cohort and needs a new revision. The direct Compose lane must match the frozen host's engine and Compose version.
  • Workloads. The actual Event Agent configuration, with a frozen shape of 14 services, 9 HTTP health checks and 2 initializers, and a pinned small fixture on oven/bun:1.4.2-slim. Both workloads are compared against all four required baselines.
  • Capacities. Native development is 4 vCPUs / 6 GiB, and the OrbStack VM is 16 CPUs / 70 GiB. Both are reported side by side and never used to scale results.
  • Cache classes, phases and scale points. warm is the only budget class. The phases are the Gate C list, including restore: an ordinary start after a retaining shutdown, with data readback. The scale points are 1, 8 and 32 branch instances and 1 and 32 MCP clients. Budgets are judged at the single-instance lifecycle. No separate numerical budget is set for the other scale points; their regressions are reported and block acceptance.
  • Run order. A 10-round Williams design for the 5 lanes, balanced for position and first-order carryover. The program floor is 5 rotated lifecycle runs.
  • Sample counts. Lifecycle metrics use the median of at least 10 runs. Tail metrics use the nearest-rank p95 of at least 60 samples, with 95% bound confidence. Statistics use admitted samples only. Flagged and failed lifecycles are counted, never rerun to fill a gap.
  • Observer overhead. A 600 s observer-only control window runs first. Observer cost is never subtracted. A difference smaller than 2 times the observer cost is inconclusive.
  • Budgets. The comparative budgets are 0.70 for footprint, 0.75 for CPU per task, and 0.80 each for warm startup and warm restart. Each uses the median and is judged per baseline, with no aggregate. The absolute budgets are info p95 ≤ 100 ms, warm admission p95 ≤ 250 ms, cached 1,000-object status p95 ≤ 150 ms, idle control plane < 1% of one core (600 s window mean), and watcher acknowledgement p95 ≤ 200 ms. As in program Gate C, only the watcher budget is limited to the small fixture.
• scripts/check-application-benchmark-protocol.py validates the frozen file, prints the schedule, and judges a retained summary with --evaluate. It exits 0 only when every verdict is pass or not_applicable, 1 on any fail or inconclusive verdict, and 2 with closed, value-free codes for an invalid protocol or summary. It never runs a runtime, calls Docker or reads host state.
• docs/performance.md records the known v5 limit the user accepted on 2026-10-19 (HACK-1123). After perf(storage): reuse unchanged storage witness proofs across warm commands #313, a warm native up takes about 17.7 s, of which about 8.7 s is witness time, against a 5 s target. These are hosted estimates. The section gives the rationale and the revisit trigger.
• CI. The runtime-models job runs the validator and the offline unittest controls.

Rework

Evaluation.

• The candidate can never be not_applicable. Such a summary is refused with exit 2 and summary_candidate_not_applicable.
• A not_applicable record must be exactly {not_applicable: true, reason}, so it cannot also carry samples. A record that carries samples is refused with summary_entry_malformed.
• A comparative budget whose baselines are all not_applicable gets an inconclusive_no_baseline verdict and is never accepted.
• Only warm summaries at the single-instance scale point are judged. A cold-cache summary, a missing cache class or another scale point is refused.
• These inputs are refused: negative samples, an expected that is not a positive integer, wrong-typed fields, unknown entry keys, and more accounted samples than planned.
• flagged and failed counts are required. A flagged sample, missing accounting, or a failed baseline lifecycle gives inconclusive_admission. A failed candidate lifecycle is fail.
• Malformed summaries no longer crash. A list-valued workload exits 2 with summary_workload_unknown; it no longer exits 1 with a traceback.

Validation. Validation now pins:

• each budget's statistic (budget_statistic_mismatch) and workload scope (budget_scope_narrowed; a scope may be broadened but never narrowed);
• the observer factor floor of 2;
• the p95 quantile, confidence ≥ 0.95, and the nearest-rank method;
• rounds_per_workload equal to the round count, and the williams design;
• workload shapes, and every required baseline in every workload's lanes (workload_baseline_missing);
• the Compose lane against the host (compose_lane_host_mismatch);
• the phases and scale points.

Duplicate budget entries are refused. A wrong-typed known_limits, workload id or budget list maps to a closed code.

Scope corrections. idle_control_plane_core_fraction and cached_status_1000_objects_ms now apply to both workloads, because Gate C limits neither to one workload. This is stricter than the first commit. Gate C's restore phase and its scale counts are now part of the frozen protocol.

Validation

• python3 -m unittest discover -s tests/python -p test_application_benchmark_protocol.py: 34 tests pass. Two of them sweep every protocol node (261) and every summary node (166), each with 13 wrong-typed values and deletion. Every result is a closed code; none raises.
• python3 -m unittest discover -s tests/python (all Python harness controls): 262 pass, 1 skipped.
• python3 scripts/check-application-benchmark-protocol.py reports valid.
• bun test tests/integration-ownership.test.ts tests/cli-spawn-env-isolation.test.ts: 16 pass, 0 fail.
• bun run --cwd packages/cli typecheck exits 0. bun run --cwd packages/cli check exits 0 with 0 errors; the 34 warnings were there before this change.
• bun run privacy:check on the staged tree: ok.
• No Rust, CLI surface or agent guidance changed.

Negative controls

• Verifier probes rerun on the new head. Each now refuses or blocks:
  • E1 (every candidate metric not_applicable), E2 (every baseline not_applicable), E3 (cold-cache summary), E4 (list-valued workload) and E8 (not_applicable over a 5× regression) are refused with exit 2, or give inconclusive_no_baseline with exit 1.
  • Negative expected values and negative candidate timings are refused.
  • Each of V1–V8 and V10–V13 is refused with a closed code: statistic changes, a baseline removed from a workload, narrowed scopes, observer factor 1, p95 method, rounds_per_workload, workload shape, Compose version drift, a numeric known_limits and a list-valued workload id.
• Mutation run. 57 mutants, one per guard, including the verifier's 19. All 57 are killed. Both of the verifier's survivors are now killed: removing the absolute max > ceiling check (killed by info_latency_ms max 1000), and removing the p95_quantile check (killed by quantile 0.9, which a 60-sample floor would otherwise satisfy). Four mutants survived the first pass: unknown scope on an extra budget, a weaker duplicate placed before the frozen entry, expected 0 with no samples, and accepting an empty verdict list. Each got a targeted case and is now killed.
• Positive control. A stricter revision still validates: a broadened watcher scope, lower ceilings, a higher observer factor and an extra scale point.

Remaining host-bound gaps

• No cohort has run. The budgets are acceptance thresholds, not results. Running the cohort is a separate, host-serialized unit on the M3 that needs exclusive runtime use.
• No harness yet produces the --evaluate summary shape, including cache_class, scale, and flagged/failed accounting. The small fixture is specified but not yet generated by a script.
• The protocol freezes the 8- and 32-instance and the 1- and 32-MCP-client scale points, but no numerical budget for them. Setting one is a goal decision, not part of this change.
• The Event Agent pin is recorded per cohort, not frozen here, because the application is private.
• The M3 host values (macOS 27.0, Docker 29.4.0, Compose 5.1.2, OrbStack 2.2.3, 16 CPU / 71680 MiB) were not re-read from the host during this rework.
• The storage-witness figures are hosted docker-e2e estimates, not M3 measurements.
• Hosted CI on the new head has not been checked yet. No Linear writes were made.

Rework 2

Commit ca810049 fixes the independent verifier's three open findings from round 1.

Oversized integers no longer crash the evaluator. Python parses a JSON integer such as 1 followed by 400 zeros exactly. math.isfinite then raised OverflowError, and the evaluator exited 1, which is the code for "judged, not accepted". Every protocol and summary number must now lie within ±(2⁵³ − 1), the interoperable JSON (I-JSON) range. The range check never converts an int to a float. An oversized sample, baseline sample, observer cost or budget value is refused as malformed with exit 2. A ratio that overflows to infinity, because a baseline is tiny but positive, is a fail and is not printed, so the report stays strict JSON. The verifier's repro now exits 2 with summary_entry_malformed.

Frozen identities are pinned. The validator now holds the exact revision-1 values and refuses any change or removal with a closed code:

• each lane's tag, version, commit, asset name and digest (artifact_identity_mismatch);
• each lane's kind, backend, engine and Compose version, plus a per-lane capacity on a baseline (lane_runtime_mismatch);
• the candidate capacity (candidate_capacity_mismatch);
• the host and engine values (host_mismatch);
• each workload's pin (workload_pin_mismatch), and any dropped frozen correctness check (workload_correctness_weakened). A revision may still add checks.

Equality is exact JSON equality, so 4.0 does not match 4 and true does not match 1. The doc test now reads the validator's pins instead of using if field in item, so a removed field is still required on the page.

Why the development profile. The protocol page now explains this choice in a new paragraph under "Capacities". At 28de2f89, the candidate's provider/profile.rs calls development an experimental allocation that cannot supply benchmark qualification, and lifecycle records carry experimental-development-not-benchmark-qualified. That label belongs to the WU02 provider comparison, whose envelope is the research profile. Revision 1 uses development on purpose for two reasons:

• Graph allocation refuses every other profile with graph_profile (provider/graph/mod.rs run and restart), so research cannot run either workload.
• Native hack up always passes --profile development (src/backends/native-project-start.ts).

A candidate record that carries the label is expected and is not a reason to refuse the cohort. Changing the label or the comment is separate runtime work. It cannot change the frozen artifact.

Validation (rework 2)

• python3 -m unittest discover -s tests/python -p test_application_benchmark_protocol.py: 37 tests pass. FUZZ_VALUES now includes ±10⁴⁰⁰, 2⁵³ and −inf in both node sweeps.
• python3 -m unittest discover -s tests/python: 265 tests pass, 1 skipped, on a serial rerun. An earlier run had 1 error while the full bun run test ran at the same time on a loaded host. I did not identify that error, and it did not recur.
• python3 scripts/check-application-benchmark-protocol.py reports valid.
• Mutation run: 11 mutants, all killed. They cover:
  • the old isfinite _number;
  • no lane identity check, no host pin, no workload pin, no correctness pin and no capacity pin;
  • always printing the ratio;
  • == instead of exact JSON equality;
  • skipping removed fields;
  • removing the profile rationale from the doc;
  • the verifier's M25 (another 40-hex candidate commit in both the JSON and the doc), which the validator now refuses with exit 2.
• gitleaks 8.30.1 over origin/next..HEAD (3 commits): no leaks. The test builds its other commit and digest values at runtime.
• bun run --cwd packages/cli typecheck exits 0. bun run --cwd packages/cli check reports 0 errors and the same 34 warnings as before.
• Full bun run test: it exited 1 locally on this M5 host at load average 14 to 17, with 7192 pass, 126 skip and 29 fail across 11 native and init files. On a serial rerun with --timeout 60000, native-compose-adoption-probe, project-env-metadata, agent-onboard-command and init-assistant passed. Five native-compose command files and two authored-recovery files still failed. Their source-CLI fixture sets a 1-second HACK_COMPOSE_STARTUP_TIMEOUT_MS. The same warm up fixture fails with private-uncertain at that timeout and exits 0 with a 30-second timeout on this host. So I classify those as host-load failures, not regressions. This PR changes no TypeScript, and hosted test passed on the previous head 89421c82 with the same TypeScript tree. Hosted CI on ca810049 is the gate for this result.
• No Rust, CLI surface or agent guidance changed.

Still open after rework 2

• No cohort has run, and no harness yet produces the summary shape.
• The M3 host values were not re-read. This session ran on a different host (Apple M5 Max, macOS 26.5.1), so the frozen M3 values could not be checked here.
• There is no numerical budget for the 8- and 32-instance and 1- and 32-MCP-client scale points. Setting one is a goal decision.
• Nobody has checked whether the harness can measure the 1,000-object cached status and the idle core fraction on Event Agent. These budgets now apply to both workloads.
• The storage-witness figures are hosted estimates, not M3 measurements.
• Follow-up candidate: the runtime-core Profile comment and the experimental-development-not-benchmark-qualified label still describe development as non-qualifying for benchmarks. Whether to relabel them is a product decision for a later candidate.
• No Linear writes were made.

Rework 2: round-2 verification (b5c2e9f8)

Round-1 findings are confirmed fixed on ca810049. I re-ran the verifier's repros on that head:

• An info-latency, baseline or observer value of 10⁴⁰⁰ now exits 2 with summary_entry_malformed or summary_observer_malformed, and no traceback.
• M25 (another 40-hex candidate commit in both the JSON and the page) exits 2 with artifact_identity_mismatch. The test suite also kills that mutant.
• The V1 and V2 weakenings are refused: a candidate or 4.x lane recast as compose, a removed asset, and a removed capacity or backend.
• The development rationale is on the page, and its source claims hold at this head (graph_profile in provider/graph/mod.rs, and --profile development in native-project-start.ts).

New in b5c2e9f8: the frozen order and sample counts are pinned. Two gaps remained after ca810049:

• Reordering the lane list and regenerating its Williams square validated, and every test still passed. --schedule and the published order table then disagreed. The validator now pins the revision-1 order exactly (schedule_order_mismatch), and a test checks the page's order table round for round.
• Sample counts were checked against the program floors (5 runs, 59 tail samples) rather than the frozen values (10 and 60). The frozen values are now the floors. A test checks that the page states both values.

Validation (round 2)

• Protocol tests: 37 pass. All 265 Python tests pass (1 skipped).
• Validator reports valid. A random fuzz of 28,000 rounds, each changing 1 to 3 nodes of both a protocol and a summary, raised nothing and produced only closed codes.
• Mutants killed: 4 of 4. They remove the order pin, the run floor and the p95 floor, and change one round in the page table.
• bun run --cwd packages/cli typecheck: exit 0. bun run --cwd packages/cli check: exit 0, with the same 34 warnings.
• gitleaks 8.30.1 over origin/next..HEAD (4 commits): no leaks.
• Trial merge with current next (099877a0): clean.
• The full local bun run test exited 1 on the M5 host, with load average 13 to 41 while other agents ran tests. I reran each of the 48 failing files in its own process with --timeout 60000. Forty-one passed outright, and mcp and test-config-compiler also passed when run again. Five native startup files still fail: native-authored-project-start, native-compose-adoption-generation, native-compose-down-command, native-compose-file-command and native-authored-project-recovery. They fail with startup-settlement and fixture-acceptance refusals, which matches the load failures in round 1. This commit changes no TypeScript, and no TypeScript test reads the changed files. Hosted test passed on ca810049 and is the gate for this result.

Still open (needs a decision or the M3 host)

These are unchanged from the earlier rework. Revision-1 cohorts, the M3 host re-read and storage-witness measurements need exclusive use of the M3 host. Budgets for the 8- and 32-instance and 32-MCP-client scale points, and the Event Agent scope for cached status and idle CPU, are product decisions. No Linear writes were made.

Hosted status on b5c2e9f8 (resumed check, 2026-10-20)

• All 16 hosted checks pass on b5c2e9f8 (run 39444856646), including test, Runtime state models, secret-scan and the four docker-e2e shards. This replaces the local bun run test load failures above as the gate for this head.
• The round-1 repros were re-run on this head and are still refused: an info-latency sample of 10⁴⁰⁰ exits 2 with summary_entry_malformed; M25, a candidate or 4.1.1 lane recast as compose, a removed asset, a changed digest, and a removed or changed candidate capacity or backend each exit 2 with a closed code.
• Trial merge with current next (b431f0e7): clean. The next changes to provider/graph/mod.rs and native-project-start.ts do not affect the development rationale, which cites the frozen 28de2f89 source.
• No new commit was needed. No Linear writes were made.

🤖 Generated with Claude Code

[linear-code]

HACK-1169