docs(research): 31 cohort test artifacts exist, 6 run in CI (#342)

[tucktuck101]

Summary

Answers #342 with the denominator criterion 3 was missing: 44 production Python modules, 13 production shell scripts and 34 YAML files under launchpad/, guarded by 31 test artifacts (18 Python test files, 13 test-*.sh). CI invokes exactly two things and covers 6 of the 31. The two unnamed gaps are the 13 shell test scripts, which no workflow runs, and review-agent's five unittest suites, which are easy to confuse with its containment controls.

Related issue

Closes #342

Issue type

Task

---

Agent provenance

Field	Value
Harness / provider	Claude Code
Model	claude-opus-5[1m]
Session reference	N/A - the harness exposes no shareable run id
Initiating human	@tucktuck101

Objective

Add launchpad/Research/342-cohort-executable-inventory.md enumerating cohort-owned executables under launchpad/ and, per test artifact, whether any CI workflow runs it.

Impacted components

launchpad/Research/342-cohort-executable-inventory.md

Approach and rejected alternatives

Enumerated from git ls-files so the method is mechanical and reproducible, then listed every run: line across all five launchpad-*.yml workflows and matched the two sets. #342 asked specifically for a reproducible mechanical method rather than a remembered list, because the failure mode is a forgotten directory.

Rejected trusting the workflow names. launchpad-review-agent-controls.yml sounds like it runs the review agent's tests; reading run_controls.py's hardcoded CONTROLS list shows it runs thirteen check_*.py scripts and none of the five test_*.py suites. That distinction is the main thing this document contributes, and a name-level reading would have missed it.

Verification

Command run:

git ls-files launchpad | ... | awk '{print $1}' | sort | uniq -c
git ls-files 'launchpad/**/*.py' | awk -F/ '...'       # prod vs test per directory
git ls-files 'launchpad/scripts/test-*.sh' | wc -l
for w in .github/workflows/launchpad-*.yml; do grep -E "^\s+run: " $w; done
sed -n '/^CONTROLS = \[/,/^\]/p' launchpad/review-agent/run_controls.py
grep -rn "test-run-guard\|launchpad/deploy" .github/workflows/

Raw output:

$ git ls-files launchpad | ... | uniq -c
  62 py
  26 sh
  34 yaml

# prod vs test per directory
launchpad/agents                   prod=3    test=1
launchpad/project-intelligence     prod=6    test=6
launchpad/review-agent             prod=24   test=5
launchpad/scripts                  prod=11   test=6

# shell per directory
launchpad/deploy                   9
launchpad/scripts                  16
launchpad/sync-labels.sh           1

$ git ls-files 'launchpad/scripts/test-*.sh' | wc -l
      13

# every run: line in the five cohort workflows
--- launchpad-adr-check.yml
python3 -m unittest discover -s launchpad/scripts
python3 launchpad/scripts/adr_boundary_check.py .
--- launchpad-issue-check.yml
(none)
--- launchpad-pr-check.yml
python3 launchpad/scripts/pr_body_check.py
python3 -m unittest discover -s launchpad/scripts -t launchpad/scripts -v
python3 launchpad/scripts/mutation_harness.py
--- launchpad-review-agent-controls.yml
pip install pyyaml
python3 run_controls.py
--- launchpad-security-audit.yml
python3 -m unittest discover -s launchpad/scripts -p "test_security_audit*.py"
python3 launchpad/scripts/security_audit.py .

$ sed -n '/^CONTROLS = \[/,/^\]/p' launchpad/review-agent/run_controls.py
CONTROLS = [
    ("check_contract.py", False),
    ("check_step2.py", False),
    ("check_step3.py", True),
    ("check_step45.py", False),
    ("check_step6.py", True),
    ("suite.py", False),
    ("check_step8.py", False),
    ("check_step9.py", False),
    ("check_step11.py", False),
    ("check_invariants.py", False),
    ("check_mutations.py", False),
    ("check_flag_guard.py", False),
    ("check_fetch_states.py", False),
]

$ grep -rn "test-run-guard\|launchpad/deploy" .github/workflows/
.github/workflows/docker.yml:142:  # a commit-pinned tag for launchpad/deploy/run.sh's guard to accept,

• Tests or checks were run and the raw output is pasted above
• The diff is confined to the scope of the linked issue
• No secrets, keys, tokens or hostnames were added to tracked files

Not verified

Whether the 13 test-*.sh scripts pass. I inventoried them; I ran none. They may be green, stale or broken, and that changes the cost of wiring them in considerably.

Whether unittest discover -s launchpad/scripts actually collects all six Python test files. It should by naming convention, but I did not run it and read the count — and #270 records that an empty discovery can read as a pass, which is exactly this hazard.

Whether review-agent's 13 check_*.py controls subsume what its five test_*.py suites assert. I established they are different invocations, not that the coverage is disjoint.

The 34 YAML files were counted, not classified. Some are deploy configuration rather than executables, and the category boundary is genuinely arguable — as it is for bin/lefthook (a cohort divergence under ADR-0017, executable, outside launchpad/) and the five ADR-0005 deployment files (cohort-owned, not executable). This counts what is under launchpad/; it does not settle the category.

Upstream's scripts/ directory is excluded, though the fork's CI depends on several of those scripts.

Security implications

None from the document. One observation for reviewers: launchpad/deploy/'s only test artifact, test-run-guard.sh, is not run by any workflow — and the deploy scripts are the code that touches the live relay host. That is the highest-consequence entry in the unrun column.

Escalations

1. prd: the cohort test suite — what we test that upstream doesn't #290's evidence needs two corrections. It says CI "invokes the cohort's checkers ... but never runs their test suites"; for launchpad/scripts the full Python suite runs on every pull request, plus mutation_harness.py. And my own earlier sweep comment on prd: the cohort test suite — what we test that upstream doesn't #290 implied the review agent's suites run in CI — they do not. Editing the PRD is not mine to do.
2. Criterion 3's scope is roughly four times what the PRD describes — 25 unrun test artifacts, with the 13 shell tests being the largest block and unmentioned anywhere.
3. The naming collision is a live trap. "review agent controls" running in CI reads as "the review agent is tested in CI". Whether the five suites should join CONTROLS, or whether the controls already suffice, is a call for whoever owns that code.

[serina-mcfall]

Blocker — the headline "31" is 32, and the note's own table says so

Using the note's own definition (a test_*.py or test-*.sh file under launchpad/), at the pinned SHA:

$ git ls-tree -r --name-only f75085d5c -- launchpad | grep -cE '/test_[^/]*\.py$'
18
$ git ls-tree -r --name-only f75085d5c -- launchpad | grep -cE '/test-[^/]*\.sh$'
14

18 + 14 = 32. The note's own coverage table sums the same way — 6 + 13 + 5 + 6 + 1 + 1 = 32. deploy/test-run-guard.sh appears as a row in the table and is then dropped from the total.

The same off-by-one surfaces twice more from the same root:

• line 49 — "26 shell scripts of which 13 are tests." Fourteen match test-*.sh; 13 is the count for launchpad/scripts alone.
• line 13 — "13 production shell scripts." 26 − 14 = 12.

Why this blocks rather than files: 31 is the PR title, the frontmatter description, and the denominator of "6 of 31" (line 90), "25 unrun test artifacts" (line 128), and the criterion-8 phrasing. Those become 6 of 32, 26 unrun, 14 shell test scripts. This is exactly the figure that gets lifted into #290 and never re-derived — and #290's criterion 8 is being written from it.

Low — .yaml vs .yml

Lines 13 and 163 say "34 YAML files". The count is right; every one of them is .yml. The label will send the next person grepping for the wrong glob.

What is right — and it is nearly all of it

The number that actually matters, "6 run in CI", is correct, and so is essentially every supporting figure. Verified independently:

• Extension totals reproduce exactly: 62 py / 26 sh / 34 yml.
• Production Python by directory reproduces exactly: scripts 11, review-agent 24 (21 + 3 in dimensions/), project-intelligence 6, agents 3 (1 + 2 in the-professor/tools/) = 44; 44 + 18 = 62. The launchpad/scripts shell figure of 16 is right too (14 in the directory + 2 in testdata/).
• The complete run: inventory across the five launchpad-*.yml workflows reproduces line for line — including the detail that the three discover invocations differ (adr-check bare at :63, pr-check with -t … -v at :100, security-audit narrowed with -p "test_security_audit*.py" at :56). That is the kind of distinction most inventories flatten.
• No workflow references launchpad/project-intelligence, launchpad/agents, or any launchpad/scripts/test-*.sh. The single launchpad/deploy mention repo-wide is docker.yml:142, and it is a comment — exactly as claimed.
• The CONTROLS list is verbatim: 13 entries, none a test module.
• suite.py references none of the five test_*.py modules.

Your "Not checked" list is honest and includes the sharpest question in the whole area — whether discover actually collects all six, given that empty discovery reads as a pass. That one is worth its own issue whatever happens to the count.

The fix here is arithmetic, not research. The research is sound.

---

Reviewed at head f75085d5c. The recount was run by me directly at that SHA.

🤖 Review drafted by Claude Code (claude-opus-5) for @serina-mcfall.