Skip to content

ci.yml: a scheduled job tests the published package as users install it - #79

Open
NSchatz wants to merge 2 commits into
mainfrom
harness/109-scheduled-published-package-test
Open

NSchatz wants to merge 2 commits into
mainfrom
harness/109-scheduled-published-package-test

Conversation

@NSchatz

@NSchatz NSchatz commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

What changed

ci.yml gains a published job that tests the caller's published package the way users install it, on the days between releases. release.yml's install gate asks that question once, right after a publish. This job asks it again on every scheduled run, because a dependency or a Node release can break an install when nothing in the caller changed.

  • When it runs. Only when the caller's workflow ran on schedule or workflow_dispatch (if: ${{ github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' }}). In a called workflow github is the caller's context. On a push or a pull request the job is skipped, so callers see a new skipped published context that no ruleset requires. It has no needs: and no continue-on-error.
  • What it does (scripts/published-check.mjs):
    1. reads the caller's own package name from its root package.json (a "private": true root fails, saying there is nothing published to test);
    2. runs npm install <name>@latest as an anonymous user in a clean room under RUNNER_TEMP, never inside the checkout. Nothing is installed in the workspace. The clean room, the anonymous npm environment, --ignore-scripts and the allow-listed child environment are imported from install-check.mjs, not copied;
    3. loads every entry point by name, once with import and once with require, each in its own process;
    4. runs the caller's synthetic smoke sample, if one is named.
  • Config packages are resolved, not parsed. An entry that is not .js/.mjs/.cjs (for example @cosyte/prettier-config's ./index.json, or @cosyte/tsconfig's ./base.json and ./library.json, which has no . export) is resolved with import.meta.resolve and require.resolve and not loaded. On Node 24.22.0, import.meta.resolve returned a URL for an export map entry pointing at a missing file, so the ESM arm also stats the resolved file.
  • New optional input published-smoke-sample (string, default ""). It names one self-contained .mjs/.cjs/.js file in the caller's checkout. The file is copied into the clean room and run there. Empty means the job only installs and imports.
  • It fails closed. Nothing is published by this job, so a red run means the package as users get it today does not work. The ::error annotation starts with npm's own error code (E404 ..., ETIMEDOUT ...), so you can tell a missing package from a registry outage at a glance.
  • Matrix. The caller's node-versions, so a broken Node release is caught. It does not use the os axis, because the clean room is built for GitHub-hosted Linux.
  • README. The ci.yml row documents the job and the input. A new section, "The scheduled check of the published package", shows a caller turning it on.

Callers. None of the library callers has a schedule: or workflow_dispatch: trigger on its CI workflow today. I read each .github/workflows/ci.yml on main on 2026-10-19. Until a caller adds one, the job is skipped for every caller. scripts/reference-publish.mjs reports the change as input-added (act: false) for published-smoke-sample and job-added (act: true) for published, which is a new check-run context.

How it was checked

test/published-check.test.mjs (25 tests) takes the step's own run: command out of ci.yml and runs it in copies of fixture callers (test/fixtures/published-check/). It runs against a registry on localhost that serves the fixtures after npm pack, so real npm installs a real tarball and Node imports it both ways. The only thing the test adds to the command is the registry URL.

  • The good caller passes, with the sample and without it (import only).
  • The caller with a deliberately wrong name fails on E404.
  • A sample that gets the wrong answer fails.
  • A misnamed sample fails before the registry is asked anything.
  • A config package passes with its JSON entries resolved, not parsed.
  • An export map pointing at a file the tarball dropped fails on import.
  • A missing JSON entry fails on both arms. Removing the ESM stat makes this test fail.
  • A clean room inside the checkout is refused before anything is installed.
  • The job's shape is pinned: the exact if:, no workspace install, the tooling checked out at job.workflow_sha, persist-credentials: false on both checkouts, and the input's type and default. Widening the if: to push makes this test fail.

The job-id and input pins in test/ci-docs-content.test.mjs, test/skipped-required-context.test.mjs and test/workflow-sha.test.mjs now include published and published-smoke-sample, recorded as additions rather than renames.

$ node --test "test/*.test.mjs"      # Node 24.22.0 and Node 22.23.3
tests 1056, pass 1056, fail 0
$ actionlint                          # rhysd/actionlint 1.7.12, all workflows
(exit 0)
$ node scripts/check-no-emdash.mjs
[no-emdash] OK: 104 tracked file(s) scanned ...

I also ran the script by hand against the live registry: @cosyte/hl7, @cosyte/tsconfig and @cosyte/prettier-config at latest (0.1.0) passed, and @cosyte/no-such-package-109 failed on E404.

One thing I found while writing the tests: the script compares real paths to decide whether it is the entry point. Comparing import.meta.url with the raw process.argv[1] treats a script run through a symlinked directory as not the entry point, so it exits 0 having checked nothing. A real checkout has no symlink, but a silent pass is the wrong failure mode.

I did not run any workflow_dispatch or scheduled run of a caller.

Harness-Task: 109

🤖 Generated with Claude Code

The new `published` job runs only when the caller's workflow ran on
`schedule` or `workflow_dispatch`. It reads the caller's own package name,
installs `<name>@latest` from npm into a clean directory outside the
workspace as an anonymous user, loads every entry point by name with
`import` and with `require`, and runs the synthetic smoke sample the caller
names in the new optional `published-smoke-sample` input (default empty:
the import only). An entry that is not JavaScript, such as a tsconfig or
prettier config JSON file, is resolved and not parsed. The ESM arm stats
what it resolved, because `import.meta.resolve` does not check that the
file exists.

`scripts/published-check.mjs` reuses install-check.mjs's clean room,
anonymous npm environment and allow-listed probe environment rather than
restating them. It fails closed: nothing is published by this job, so a red
run is the finding, and the annotation leads with npm's own error code.

test/published-check.test.mjs runs the step's own command from ci.yml in
copies of fixture callers against a registry on localhost: the good caller
passes, a deliberately wrong name fails on E404, a failing sample, a dropped
ESM build and a missing JSON entry fail, and a config package passes with
its entries resolved. The job-id and input pins in three suites record
`published` and `published-smoke-sample` as additions. No caller has a
`schedule` trigger today, so this changes no caller's runs until one adds
it.

Harness-Task: 109

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread scripts/published-check.mjs Fixed
- Escape backslashes before pipes in the step summary's table cells
  (js/incomplete-sanitization). A `\|` already in a check's detail would
  otherwise end the cell early. A test now pins the escaped form.
- Run the probes and the copied smoke sample by constant file names with
  the clean room as `cwd`, so no path built from `RUNNER_TEMP`, `tmpdir()`
  or the command line is passed to a child process as an argument
  (js/indirect-command-line-injection and
  js/shell-command-injection-from-environment, both reported at the
  shared runner's `spawn` in install-check.mjs).
- Tell sample authors to prefer `.mjs` or `.cjs`. The clean room has no
  `"type"`, so a `.js` sample in ESM syntax relies on Node's module
  detection (a reviewer note).

Harness-Task: 109

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants