Skip to content

Fast tests: CI names the file a change drops, and the hand-kept count is gone - #764

Draft
sdcarlson wants to merge 3 commits into
mainfrom
claude/charming-hawking-mq9fh4
Draft

sdcarlson wants to merge 3 commits into
mainfrom
claude/charming-hawking-mq9fh4

Conversation

@sdcarlson

@sdcarlson sdcarlson commented Oct 31, 2026 •

Copy link
Copy Markdown
Collaborator

What

vitest.fast.config.js held export const FAST_TEST_FILES = 169;, and the required CI check failed whenever the fast run collected a different number of files. So every pull request that added, moved or deleted a test in a listed directory had to change that one line, and parallel pull requests conflicted on it by construction. #548 created the file on 2026-10-08 and 44 commits have changed it since, and the open #762 and #763 both change the line (to 170 and to 174), so whichever merges second conflicts with the first.

The count stood for one fact: what the fast run collected before this change. Git holds that fact, so the check now computes it and names the file instead of comparing a number:

  1. Every entry in FAST_TESTS matches a file the run collected. This is the [HIGH PRIORITY · Fable] RISE architecture redesign: three constraints, five layers, five rooms #377 failure mode: [HIGH PRIORITY · Fable] RISE architecture redesign: three constraints, five layers, five rooms #377 renamed or moved three listed files, their filters matched nothing, and Vitest ran without them until Run the three unit tests that #377 moved #421 noticed.
  2. A test the run collects that this change renames or moves is still collected where it lands. git diff --find-renames HEAD^ HEAD lists the moves. On a pull request HEAD is GitHub's merge commit and HEAD^ is the base branch. This catches a test moved out of a listed directory (src/core/decision/**, src/live/adapters/** and the rest), which no list entry can see.

The CI job's checkout now fetches two commits (fetch-depth: 2) so that HEAD^ exists. Nothing else in the job depends on the depth: the recitation hydrate fetches its own ref, and the diagram and the arena read HEAD, git status and git ls-files.

AGENTS.md, docs/RELEASING.md and the config's header said the count existed. They now say what CI checks. 4 files, +27 −13. (The second and third commits are a demonstration and its revert; see Verification.)

Shapes considered and not taken

  • Every *.test.js under the listed directories was collected. For the nine glob directories, Vitest already guarantees this, so the check would add nothing. For the 15 directories of the 87 explicitly listed files, it is false today: 343 tests outside the fast run sit directly in them (438 anywhere under them), 149 of them in src/core.
  • Every test file is either fast or in a sorted slow-only list. The list would start at 483 entries, one per unit test outside the fast run. Every new or moved test outside the fast directories, which needs no edit today, would then have to edit it.

Failure modes

Each row is one pull request's worth of change on top of this branch. Both checks read the same Vitest report.

Change Files collected main's count check This check
None (this branch as it is) 169 pass pass
Add a test under a listed directory (src/core/decision/) 170 fail: collected 170, expects 169 pass, with no edit
Add a test and list it, as #762 and #763 do 170 fail until the count changes too pass: the entry is the only edit
#377's rename in reverse: src/components/Home.test.js → Portal.test.js 168 fail, names no file fail: vitest.fast.config.js lists src/components/Home.test.js, which matches no file the fast run collected. and src/components/Home.test.js moved to src/components/Portal.test.js, which the fast run does not collect.
#377's move in reverse: src/components/read/Chamber.jev-look.test.js → src/components/ 168 fail, names no file fail: the same two lines for that file
Move a test out of a listed directory: src/core/decision/variance.test.js → src/core/ 168 fail, names no file fail: src/core/decision/variance.test.js moved to src/core/variance.test.js, which the fast run does not collect.
Rename out of the glob's pattern: src/live/adapters/mock.test.js → mock.spec.js 168 fail, names no file fail: names the file and where it went
Rename a listed directory: src/live/hosts/ → src/live/hosting/ ¹ 165 fail, names no file fail: names the entry src/live/hosts/**/*.test.js and each of its four files
Delete a listed test: src/visuals/attractor.test.js 168 fail, names no file fail: names the entry
List a file that does not exist (worker/count-typo.test.js; on main, with the count raised to 170) 169 fail: collected 169, expects 170 fail: names the entry
Move a listed test into a listed directory: src/core/kev-client.test.js → src/core/decision/ 169 pass fail: names the stale entry src/core/kev-client.test.js
Delete a test under a listed directory: src/core/figures/brands.test.js 168 fail pass (see below)
Move a test between listed directories: src/live/host/ → src/live/hosts/ 169 pass pass
Move a test outside the fast set 169 pass pass

¹ The scenario did not fix the code that imports from src/live/hosts/ (the Worker's mcp-server.mjs, the Changes pipeline and the Live host), so 20 of the 165 collected files also failed. A real rename fixes those imports; the check's verdict does not depend on them.

Still caught, and now each failure names the file: a listed test renamed or moved (the #377 mode), a test moved out of a listed directory, a rename out of a glob's pattern, a listed directory renamed or emptied, a listed test deleted, and an entry that names no file.

Newly caught: a listed test moved into a listed directory leaves a stale entry. The count could not see it, because the number did not change.

No longer caught:

  • Deleting a test under a listed directory. The count failed on it; this check does not. The deleted test is in the diff, and the count's version of the check was a number bump.
  • Dropping a test from the run on purpose. Removing its entry, or adding it to the exclude list, also needed the count changed. Now the edit to vitest.fast.config.js is the whole change, and it is in the diff.
  • A move git does not pair as a rename. That happens when more than half the file changed in the same change (--find-renames defaults to 50% similarity), or when a very large diff exceeds git's rename limit. It reads as a deletion plus a new file outside the fast set. The first check still catches it for a listed file; for a file under a listed directory, nothing does.
  • Several commits pushed to main at once. Only the last commit's moves are compared. Pull requests are compared on GitHub's merge commit, which covers the whole pull request.

Verification

In this PR's CI (the required CI check):

  1. 4016eb9, the change: passed in 7m04s. The log shows the checkout fetching --depth=2, HEAD is now at e1a1eea Merge 4016eb9… into 19aabc7…, and Test Files 169 passed (169).
  2. e5770f3, the demonstration ([HIGH PRIORITY · Fable] RISE architecture redesign: three constraints, five layers, five rooms #377's rename in reverse, plus a correct move out of src/core/decision/ with its import fixed): failed at the fast-test check, after Test Files 167 passed (167). The check printed:
    vitest.fast.config.js lists src/components/Home.test.js, which matches no file the fast run collected.
    src/components/Home.test.js moved to src/components/Portal.test.js, which the fast run does not collect.
    src/core/decision/variance.test.js moved to src/core/variance.test.js, which the fast run does not collect.
    
  3. 2e76701 reverts it, back to the same tree as 4016eb9: passed in 6m27s, browser gate included. A squash merge leaves only the change.

GitHub's merge commit for this PR (e1a1eea) has 19aabc7 (main) as its first parent and 4016eb9 as its second, so HEAD^..HEAD there is exactly this PR's diff.

Locally (Node 22.23.4, Vitest 4.1.12), the table above. Each row:

  • branches from 4016eb9, makes the change and commits it;
  • runs the fast suite exactly as ci.yml runs it (npx vitest run --config vitest.fast.config.js --reporter=default --reporter=json …);
  • runs the check, extracted verbatim from that commit's ci.yml, with HEAD^ as the base;
  • runs main's count check, from main's ci.yml, on the same report.

The check takes about half a second. Also clean: node scripts/ci-hygiene.mjs, npm run roadmap:check, npm run docs:diagram (no diagram change), the wiki build (which also passed here as Build and publish), and src/core/security-policy.test.js, which reads ci.yml. The YAML parses and the step passes bash -n. shellcheck is not installed in this environment.

The open pull requests that change the count

#762 and #763 change the line this deletes. Once this merges, each will conflict there once. The fix is to drop the line and keep their new entries, which need nothing else.

Not done

  • No task record in docs/product/tasks/. This is a CI-only change with no tracked task. Say if you want one in the foundation lane.
  • An independent review, before this leaves draft. This session wrote the change and does not review it.

Class: auto
Agent-platform: claude
Gates-affected: none

🤖 Generated with Claude Code

https://claude.ai/code/session_01VHrJ4j5ztkHoJ9z9dkLMCv

The required CI check compared the number of files the fast unit run
collected with FAST_TEST_FILES, a number every pull request that added,
moved or deleted a test in a listed directory had to edit. Parallel pull
requests collided on that one line by construction.

The check now computes what the count stood for, and names the file:
- every entry in vitest.fast.config.js must match a file the run
  collected (the #377 failure mode: an entry left behind by a move);
- a test the run collects that this change renames or moves (git's
  rename detection over HEAD^..HEAD) must still be collected where it
  lands (a move out of a listed directory).

The checkout fetches two commits so that HEAD^ exists: on a pull request
it is the base branch under GitHub's merge commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VHrJ4j5ztkHoJ9z9dkLMCv
@netlify

netlify Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

❌ Deploy Preview for rise-v2-symbolic-experience failed.

Name Link
🔨 Latest commit 2e76701
🔍 Latest deploy log https://app.netlify.com/projects/rise-v2-symbolic-experience/deploys/6ae650a67f3f5dd85bb6bc86

#377's own rename in reverse (src/components/Home.test.js to
src/components/Portal.test.js), and a correct move out of a listed
directory (src/core/decision/variance.test.js to src/core/variance.test.js,
its import fixed). Neither file is collected where it lands, so CI must
fail at the fast-test check and name both. The next commit reverts this.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VHrJ4j5ztkHoJ9z9dkLMCv
CI run 41239062776 failed at the fast-test check on the merge commit of
e5770f3, after the fast run collected 167 files and all passed:

  vitest.fast.config.js lists src/components/Home.test.js, which matches no file the fast run collected.
  src/components/Home.test.js moved to src/components/Portal.test.js, which the fast run does not collect.
  src/core/decision/variance.test.js moved to src/core/variance.test.js, which the fast run does not collect.

This reverts commit e5770f3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VHrJ4j5ztkHoJ9z9dkLMCv

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants