Friction report from a 20h multi-lane session: 24 measured defects, and every one is the same shape — something reporting success while doing nothing

[jobordu]

Relayed from TEAMLEAD-core after a ~20-hour multi-lane session on Borduas-Holdings/Blazing-Back. The measurements are theirs unless marked [DX corroborated], where I reproduced the symptom independently in the same session.

---

⭐ The meta-point, relayed verbatim in substance

Every one of today's defects — mine and the lanes' — was the same shape: something reporting success while doing nothing. A reaper firing 8 times and reaching nothing. A rule merged with zero callers. A caller pinned before the rule existed. An image correctly digest-pinned to a deprecated runner. 564 tests executed by nothing. A test asserting nothing and passing. Twice, a workflow that could not parse reading as CLEAN.

The ask is not "more checks" — it is that every check should be able to distinguish PASSED from NOT RUN, and today most of them could not.

That single sentence is the highest-value line in this report and should drive the prioritisation below.

---

A. Instruments that report clean while blind — the expensive class

A1. gh pr checks parsed output hides non-required failures. Filtering the table on $2=="fail" returned empty while /commits/{sha}/check-runs reported a FAILURE. Audited across 8 merges: the miss was 8 of 8. ⇒ Never parse gh pr checks; use the check-runs API or GraphQL statusCheckRollup.

A2. A commit STATUS is not a CHECK-RUN, and the two endpoints do not union themselves. tier-coverage is published as a commit status and does not appear in /commits/{sha}/check-runs at all. A predicate built on check-runs alone sees 5 of 6 required contexts and reads the sixth as "hasn't run".

[DX corroborated — partially] On origin/main I measured 26 check-runs, tier-coverage among them: 0. ⚠ I could not corroborate the other half on that SHA — main carried 0 statuses, so "present in /status" is unverified by me. The absence from check-runs is confirmed; the union requirement follows from the design (the gate publishes a status precisely because the Status API enum has no skipped).

A3. An unparseable workflow produces ZERO check-runs and reads as CLEAN — not red, absent. Hit twice (unquoted ! in an if:; a duplicate needs: key). Tell: the run is listed by its file path instead of its name:, with no jobs and failed to get run log: log not found. ⇒ The only reliable check is "did the workflow START on this SHA" — actions/runs?head_sha=… filtered by name, length >= 1. Cheap proxy: context count (57 = ran, 4 = didn't).

A4. yaml.safe_load silently accepts duplicate keys and keeps the last; GitHub Actions rejects them. A merge resolution left two needs: in one job — the file parsed, reported 42 jobs with correct values, and GitHub refused it.

[DX corroborated] yaml.safe_load("a: 1\na: 2\n") → {'a': 2}. Silent, last-wins. A duplicate-key-detecting Loader is ~one screen and TEAMLEAD offered to supply it.

---

B. Shell / environment traps

B1. zsh parses "$VAR:path" as a history modifier and eats part of the path. ⚠ With a tag it does not error — git show succeeds on the bare ref and prints the commit diff, so a downstream grep -c returns confident wrong numbers. A 7-row version table was built this way; 4 rows were accidentally right and the errors cancelled, so it read as coherent. ⇒ Always ${VAR}:path, and assert git cat-file -t is blob.

[DX corroborated — six times in one session] :c (from control-plane/…), :s (from scripts/…), :e (from e2e/…), :t (from tests/…). Twice it threw fatal:; four times it returned a silent 0 that read as a finding — including node_pool_tier= → 0, which looked exactly like "the code was refactored away" when the truth was 3 call sites. Same file, same commit: inline 0 lines, R="$M:$P" 701 lines.

B2. zsh does not word-split $VAR; set -- $SPEC yields empty positional args. Needs ${=VAR}. [DX corroborated] — a multi-file loop ran once on a newline-joined blob and returned 0 for a file I had measured minutes earlier.

B3. $? after a pipe is the last command's status. [DX corroborated ×3] — including install rc=0 that was tail's, and exit: 0 on a kubectl call that had actually failed auth.

B4. Foreground sleep is blocked by the harness, and sleep 120 inside a Bash call hits the 2-minute ceiling (exit 143). ⇒ No clean wait primitive; polling costs GitHub quota shared with the CI provisioner.

B5. src-layout repos fail pytest from root (ModuleNotFoundError) without PYTHONPATH=src. ⚠ Looks identical to a real import bug; nearly written off as "env issue" twice.

B6. Local ruff version mismatch wanted to reformat 27 files the repo disagrees with — nearly shipped inside a 2-line fix. [DX corroborated] — this box has two rufffs: bare ruff is 0.15.20 (the CI pin) and python3 -m ruff is 0.15.12. They answer different questions.

B7. ruff check and ruff format --check are two separate gates; passing one says nothing about the other. Hit twice today by two different agents. [DX corroborated]

---

C. gh CLI behaviour

C1. Logs are unavailable while a run is in_progress — "logs will be available when it is complete" — even for a job that has already failed. You cannot diagnose a failed job until every sibling finishes. Cost ~40 minutes of blocked diagnosis. [DX corroborated] — I hit this on a B1b failure while E1 held the run open.

C2. gh run rerun --job <id> is rejected while the run is in_progress. A 55-minute E1 blocks retrying an unrelated failed job.

C3. gh run rerun prints nothing on success — indistinguishable from a no-op.

C4. gh run rerun --failed re-runs CONSUMERS but not PROVISIONERS. On an ephemeral-runner-pool workflow this re-runs jobs against a pool that is already spent. Safe only when the job is runs-on: ubuntu-latest, which must be checked first.

C5. gh api .../actions/jobs/{id}/logs returned 0 bytes where gh run view --job <id> --log worked.

C6. GraphQL node limits fail silent-ish: 25 PRs × 60 contexts × 20 threads is fine; 60 × 90 × 40 returns an error with no data key, and a naive parser raises KeyError instead of surfacing the message. ⇒ Always print errors[] before indexing data.

C7. gh api … | head -c N truncates via SIGPIPE and produces "corrupt JSON" that is nothing of the sort. Write to a file, then read.

---

D. Fleet / coordination

D1. ⛔ The SendMessage roster carried only 4 of 10 live lanes. PRODUCT-core and DEV2/4/5/6-core were absent from ListAgents while alive and healthy in Daintree. A message to PRODUCT-core silently went nowhere and was only discovered because the operator noticed. ⇒ A send to an unresolved name must fail loudly, not resolve to nothing.

D2. Daintree terminal.sendCommand timed out twice mid-dispatch, retriable but with no delivery confirmation — fire-and-forget, no way to know a lane received a brief.

D3. Lanes share the project memory directory, and writes are being lost.

[DX corroborated, and worse than reported] I found 6 of 7 memory entries I wrote today were unindexed — the body files existed, the index pointers did not. The single survivor was the only one where I replaced an existing line rather than inserting a new one.

Two compounding causes: (a) the index is live-contended — between reads seconds apart it went 21,446 → 20,747 → 22,195 bytes with the mtime moving; (b) it is ~5KB over its own documented ~17KB load cliff, after being archived today specifically "to fit" — so the newest entries sit past the truncation point and do not load.

⚠ The generalisable part: my write-then-verify pattern verified at write time, and the entries were gone hours later. Verification at write time does not establish durability under a concurrent writer — which is the report's own meta-point applied to the memory substrate.

D4. One lane (codex-based) has no Daintree MCP at all — it can only hand back via files under .planning/operator/. Its results are invisible unless you know to look.

D5. The Stop hook replayed a stale handback ~10 times — one lane's cached identity message, containing three already-refuted claims, re-delivered every turn with no way to clear it.

---

E. Safety-flag parsing — the worst near-miss

E1. ⛔ DRY_RUN=1 EXECUTES A LIVE DESTRUCTIVE PASS. The guard is os.environ.get("DRY_RUN","false").lower()=="true" — so "1" is false. TEAMLEAD put that flag in the one instruction whose entire purpose was to make a destructive step safe; a lane caught it before it executed. Filed as Borduas-Holdings/Blazing-Back#1581.

E2. --dry-run on the same script is silently ignored. ⇒ Both obvious ways to ask for a preview fail, in two different ways, and neither says anything.

E3. The mirror image: [ -n "$VAR" ] and ${VAR:+…} treat the string "false" as truthy. A dry-run dispatch sets "false"; a scheduled run sets empty. So a presence test fires on the dry run and turns it live. ⇒ The guard must be [ "$VAR" = "true" ].

⇒ E1 and E3 together are the sharpest illustration of the meta-point: two safety flags, both parsed by presence rather than value, each failing in the direction that runs the destructive path.

---

F. Asks, in priority order

1. A merge-ready check evaluating all four limbs, unioning BOTH status endpoints: required contexts green (ABSENT ≠ SKIPPED ≠ green), zero unresolved threads, every non-required failure listed by name, and the workflow actually started on this SHA.
2. A duplicate-key + unquoted-! YAML linter for workflow edits, run pre-push.
3. A "did this rule ever fire?" retrospective harness — run a new conformance rule against the pre-fix state of the artefact it was written for. A rule was merged that reported NOT APPLICABLE on the exact repo it was built for; caught only by asking "would this have caught the thing it was built for?" after merging.
4. Loud failure on unresolved agent names in SendMessage.
5. A wait primitive that consumes no API quota and does not hit the 2-minute Bash ceiling.

[jobordu]

Addendum — two items measured after the original report, both independently corroborated

A5. /actions/runs/{id}/jobs returns only the LATEST ATTEMPT, and total_count reports the filtered count as a total

Rerunning a single job creates a new attempt and replaces the visible job list. Nothing in the response says so.

[DX corroborated] on the cited run 32851921712:

run_attempt                     2
default   /jobs                26 jobs   total_count = 26
?filter=all&per_page=100       69 jobs   total_count = 69
response keys (default)        jobs, total_count      <- no filter indicator

⭐ The sharpest part is total_count. A field named total reports 26 on the default view — the count of the filtered subset, presented as the total. So the obvious defensive check ("did I get all of them? compare len(jobs) to total_count") agrees with itself and is wrong. This is the same family as a pagination.total that echoes the page limit: the field that exists to detect truncation is itself truncated.

⇒ Cost: two lanes reported contradictory job lists for the same run id and both were correct — one read attempt 1, the other attempt 2 after a rerun. It cost a round trip and a mutual accusation of misreading.

⇒ Ask: any helper asserting "what a run did" must pass ?filter=all; a default-view read must be labelled as a claim about the latest attempt only; and total_count must not be used as a completeness check on this endpoint.

⚠ Same shape as A1 and A2: an instrument answering a narrower question than the one asked, without saying so.

---

D3 (extended). "Indexed" and "will load" are DIFFERENT PROPERTIES — and verifying the first tells you nothing about the second

The original report noted lost memory writes. Two independent readings now sharpen it, and they disagree in a way that turns out to be the finding:

• TEAMLEAD-core: 13 of 13 memories written today are present and indexed. Write-then-verify held.
• DX-core: 6 of 7 were unindexed hours after writing. Write-then-verify did not hold.

⇒ Both are true. The difference is insertion position, not correctness of the write:

MEMORY.md          24,246 bytes
documented cliff  ~17,408 bytes
over by            ~6,838 bytes        (~126 index lines; the tail never loads)

[DX corroborated] After re-adding three entries at the top of the first section, their byte offsets are 295 / 495 / 792 — all above the cliff, so they load. TEAMLEAD's 13 survived for the same reason: they were inserted at the top of the list.

⇒ ⭐ An indexed entry past the cliff is invisible on the very next session — immediately, not eventually. So there are two distinct failure modes with one symptom:

1. Lost pointer — the entry is not in the index (concurrent writer clobbered the insertion).
2. Present but past the cliff — the entry is indexed and still never loads.

Verifying (1) does not detect (2). A writer that checks "is my line in the file?" passes in both cases.

⚠ And the file was archived today, specifically to fit the cliff, and is already ~7KB over. Manual archiving is not keeping pace with the write rate.

⇒ Ask: the memory writer should (a) report the index's size against the cliff after every write, (b) refuse or loudly warn on a write that lands past it, and (c) archive automatically rather than depending on someone remembering. Position within the index is currently load-bearing and completely undocumented.

⚠ This is the report's own meta-point applied to its substrate, in both directions: verification at write time proves neither that the state holds later, nor that it holds at all.

[jobordu]

Addendum 2 — and A6 is the example this whole issue was arguing for

Measurements below are TEAMLEAD-core's except where marked [DX corroborated].

---

⭐ A6. The hardening tool reports skipped on a repo with 375 passing tests

nf:harden was pointed at akash-github-runner — a good instinct, since two conformance rules had just been proven vacuous. Measured:

JS/TS test files found : 0
python test files      : 26
passing tests on main  : 375
package.json           : ABSENT

⇒ Result: HARDENING SKIPPED — no test files found, return status skipped.

[DX corroborated at source] — ~/.claude/nf/workflows/harden.md:

:39  find "$SCOPE_ROOT" \( -name "*.test.js" -o -name "*.test.cjs" -o -name "*.test.ts" \
:40    -o -name "*.spec.js" -o -name "*.spec.ts" \)          <- zero Python patterns
:45  If $TEST_FILES is empty:
:48    nForma ► HARDENING SKIPPED — no test files found
:54  STOP — return status: `skipped`
:60  fallback $RUN_CMD = 'npm test'                          <- wrong runner for these repos

Why this is the issue's own thesis, inside the tooling

Not red. Not an error. skipped — which reads as "nothing to do here".

A tool built to find weak tests cannot see 375 of them, and reports a clean neutral outcome while structurally blind to what it was pointed at.

⇒ And the message is the precise failure: "no test files found" is true of the patterns tried and false of the repo. It answers "did my five globs match?" while appearing to answer "does this code have tests?" — the same shape as A1, A2 and A5, in the tool whose purpose is to catch exactly this.

What it needs, in priority order

1. Python discovery — test_*.py / *_test.py, with $RUN_CMD = pytest when pyproject.toml or pytest.ini exists. The package.json → npm test fallback is also wrong for these repos, not merely unhelpful.
2. ⛔ A skip must not be silent. A skip caused by nothing matching must print which patterns were tried against which root. As written, "no test files found" is indistinguishable from "this code needs no tests" — and that distinction is the entire point of this report.
3. For GUARD code, adversarial pressure is not boundary values — it is the REVERT-PROBE: revert only the fix and confirm the new test goes RED. That single discriminator caught both vacuous rules. A guard that passes both before and after the fix is the vacuous case, and the current categories (boundary / null / coercion) would not surface it.

⚠ This is not "nf:harden is broken." The workflow's logic and convergence design are sound; it is scoped to JS/TS and was pointed at Python. The defect is that the scope mismatch produces SILENCE instead of a diagnosis.

---

A7. A shared checkout produced a false "main is broken" reading — third time in one session

pytest in the shared checkout returned 2 failed, 356 passed, one step from a report that a merge had broken main. The checkout was on another lane's branch with 2 uncommitted files. An isolated worktree at origin/main: 375 passed, clean.

⇒ Third false reading from the shared checkout in one session — the others being a stale local branch that made a merged fix look absent, and a worktree collision on a branch another lane held.

⇒ Ask: any helper that runs tests or reads files to support a claim about a ref must use an isolated worktree at that ref, never the working tree. Cheap, and it removes the whole class.

---

A8. A hardcoded pattern list IS the tool's coverage, and it is invisible at the call site

Generalising A6. The same shape has now appeared three times in this repo family:

• nf:harden's five JS/TS globs — 0 of 26 Python test files (A6).
• guard_required_check_placement.py's REQUIRED_PREFIXES tuple — covered 3 of 5 required contexts for months (#1133).
• A conformance rule requiring github-runner: with a tag — the tagless form was never examined (A vendored goal file is specimen and doctrine at once, with no discriminator — three of four goals are scoped to another product and the presence test passes anyway #16).

⇒ In each case the list is the scope, the scope is undocumented at the call site, and what was skipped is never reported.

⇒ Ask: wherever a hardcoded pattern list defines scope, the tool must report what it matched AND what it skipped, not merely act on the matches. A coverage claim from a pattern list is only as good as a list nobody has read.