Skip to content

fix(native): keep Compose-only projects on the Compose backend with closed refusals - #365

Merged
roodboi merged 4 commits into
nextfrom
wf/g2-native-compose-feature-refusals-b21
Oct 22, 2026
Merged

roodboi merged 4 commits into
nextfrom
wf/g2-native-compose-feature-refusals-b21

Conversation

@roodboi

@roodboi roodboi commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

User decision (2026-10-19): in v5, unchanged real Compose projects stay on the supported Compose backend. Native support for these features is post-v5 work (HACK-1270). This PR makes native hack up and hack restart refuse each blocking Compose feature exactly, before any mutation. It also documents the features and adds a hack doctor check that reports them.

The explicit HACK_NATIVE_ADAPTATION path stays supported. A network that a valid adaptation lists in isolatedNetworks is not treated as an external network: the check classifies the Compose file after that adaptation, as native start does.

Root cause

Native image-project up (handleNativeUp → startNativeProject) had no early classification of Compose-only features:

  • External networks and non-arm64 platform were not checked on the TypeScript side. They failed only after runtime check-project-share, run-storage preparation, lifecycle before hooks and runtime up (the runtime-core review rejects external_network, and registry admission selects arm64 only).
  • Builds and labels were refused, but only as the general NATIVE_CONFIGURATION_REFUSAL. That came after the source preflight and storage preparation, and it did not say the project belongs on the Compose backend.

Rework after verification (adaptation regression)

The first version of this PR (9ce0f3b) read only the original .hack/docker-compose.yml. It ignored the documented HACK_NATIVE_ADAPTATION isolatedNetworks selection, which native start applies to the normalized copy (replaceNetworks sets x-hack-isolated). So "isolatedNetworks": ["hack-dev"] with an external hack-dev network was refused with the Compose-backend line before the adaptation was read. That blocked up, restart and restart --recover, and native doctor warned about it. On next the same project passes native review and reaches runtime check-project-share and runtime up.

Fix (76003b5):

  • findComposeBackendFeatures(compose, {isolatedNetworks}) does not count a network named in the set as external. Other features are unaffected.
  • readComposeBackendFeatures(composeFile, {adaptationFile}) reads the selection only when the unadapted file has an external network. A project without one keeps its existing refusal order.
  • readAdaptedIsolatedNetworks reads the file with native start's own readNativeSelection (absolute path, bounded, unchanged regular file). If the file cannot be read, the selection is not version 1, or isolatedNetworks is not a bounded list of unique strings, it refuses with the same one-line E_CONFIG_INVALID message native start gives, before any runtime request. Native start still validates the whole selection against the normalized Compose file, including that each listed name is an external network.
  • handleNativeUp (covers up, restart and restart --recover) passes process.env.HACK_NATIVE_ADAPTATION, the same value it gives startNativeProject. Native doctor passes the same value. Compose doctor does not, because the adaptation is native-only.
  • Docs: the "External network" row, the refusal-contract paragraph and the doctor paragraph in docs/guides/native-candidate.md now describe the adaptation. The adaptation section links back to the Compose-backend section, and docs/reference/native-config-import.md qualifies its summary. The two sections no longer contradict each other, and isolatedNetworks is not retired.

Fix (unchanged from the first version)

  • src/lib/native-compose-backend-features.ts classifies the project's Compose file into a closed, ordered set: external_network, reserved_label, unsupported_build, foreign_platform.
    • External network: top-level external: true or external: {name}, unless it is x-hack-isolated: true or isolated by the adaptation.
    • Reserved ownership labels: keys starting with com.docker.compose., io.hack-local. or io.hack.native-config., on service, network and volume labels. Routing (caddy*), cache (hack.dependencies.*), one-shot and other ordinary labels keep their native behavior.
    • Build: reuses the import preview's mapLegacyComposeBuild. The result matches what hack config import --dry-run reports as invalid_or_unsupported_build or image_build_exclusive.
    • Platform: anything other than linux/arm64 or linux/arm64/v8.
  • handleNativeUp calls refuseNativeComposeBackendFeatures after option refusal and project selection, before branch, config, env-fingerprint, lifecycle, shared-source, storage and runtime work. --detach still gets its one-line E_USAGE first (fix(cli): pass child flags through and print closed native refusals #359). down is not affected. Authored and adopted native-config projects go through tryNativeComposeCommand first and never reach this check.
  • Each refusal is a HackCliError with the stable code E_NATIVE_PROJECT_UNSUPPORTED and one value-free line that names every feature found and the HACK_RUNTIME_BACKEND=compose selection. detail is {backend: "compose", features: [...]}.
  • Unreadable, oversized (over 256 KiB), symlinked or unparseable Compose files produce no features. Native input review still refuses them as before.
  • hack doctor has a compose backend features check, in both native and Compose doctor. On native, a feature found is a warn. On Compose it is informational (ok). With --json, the check carries composeBackend.features (closed names only).

The CLI surface (options and help) is unchanged, so docs/reference/cli.md was not regenerated.

Tests

  • tests/native-compose-backend-features.test.ts (unit, 13 tests). The original cases are kept. New in the rework:

    • An adaptation-isolated network is not external. A parity check runs the real adaptNativeProject on the same selection and classifies its normalized output the same way. A network that is not listed is still external, and isolation never hides another feature.
    • A file classified with a valid isolating selection has no features, while a selection that lists another network or has no isolatedNetworks leaves the external network.
    • Five invalid selections refuse with E_CONFIG_INVALID and the exact message. For each one, the real adaptNativeProject throws the same message. Without an external network the selection is not read. Missing and relative paths refuse.
  • tests/native-compose-backend-features-cli.test.ts (real CLI through privateCliEnv, stub runtime, 15 tests). The original cases are kept. New in the rework:

    • The verifier's repro: external hack-dev + {"version":1,"isolatedNetworks":["hack-dev"]} + HACK_NATIVE_SHARED_SOURCE=1. hack up passes the check and reaches runtime check-project-share and then runtime up, as on next.
    • hack restart with the same selection reaches restart preflight's own Native restart has no existing run line.
    • A selection that isolates a different network still gets the Compose-backend line, with 0 runtime calls and an unchanged tree.
    • An invalid selection (version 2) gets one E_CONFIG_INVALID line, with 0 runtime calls and an unchanged tree.
    • Native doctor --json with the selection reports ok and features: [].
  • Mutation proof for the rework. Each mutation was applied, both files run, and the change reverted with git checkout:

    Mutation Failing tests
    handleNativeUp does not pass the adaptation 3 (up, restart, invalid selection)
    Classifier ignores the isolated set 5
    Selection read even without an external network 1
    isolatedNetworks shape not validated 1
    version not validated 2
    Native doctor does not pass the adaptation 1

    The first version's mutation results (removing the call, each rule, doctor wiring) still apply to the unchanged cases.

Evidence

  • Bun 1.4.2 (mise x bun@1.4.2), macOS arm64 host. Load average was between 30 and 95 during these runs.

  • bun run --cwd packages/cli typecheck: 0 errors. bun run --cwd packages/cli check: 0 errors and 34 existing warnings. biome check on the changed files is clean.

  • Focused and required: the two PR test files, tests/cli-spawn-env-isolation.test.ts, tests/cli-reference-docs.test.ts, tests/integration-ownership.test.ts and tests/native-refusal-cli.test.ts: 63 pass, 0 fail.

  • tests/doctor-*.test.ts, tests/native-project-adaptation.test.ts and tests/native-config-import*.test.ts: 539 pass, 3 fail. The 3 failures are the capture process-group tests in tests/native-config-import-build.test.ts (about 505 ms each). The verifier found the same failures with origin/next source, so they come from this host, not this PR.

  • After bun run build and bun run build:config-compiler, all six scripts/check-native-*-plan-cli.ts exit 0. No native refusal code or message changed. The E_CONFIG_INVALID adaptation message is native start's existing one.

  • gitleaks 8.30.1: no leaks. The scan ran as gitleaks dir on a git log -p origin/next..HEAD export, because the worktree sandbox blocks gitleaks git.

  • Full bun run test: exit 1 after 1 h 27 min on the loaded host. 114 files had failures in the parallel run. A serial rerun of each with --timeout 60000 left 20 files failing. 19 are authored/adopted native-compose suites, which never reach handleNativeUp's check, and their failures are fixture timeouts and unsettled children. The 20th, native-env-metadata, passed 32/32 on a later rerun with this PR's code. As a baseline, src/commands/project.ts and src/commands/doctor.ts were swapped to the merge base ec223cdb. native-compose-down-command (1/13), native-compose-storage-fence-cost (0/4), native-compose-retained-storage-command (0/5) and native-authored-process-incarnation (9/1) failed the same way, so these failures are inherited under this load.

  • The adaptation CLI case also accepts a timed-out or unreadable stub reply (94e12a9). Under load, both appeared in the parallel run, and each is reachable only after the check passed. The original control: ordinary labels… case is sensitive to the same load in the parallel run and passes serially.

  • Hosted CI on 76003b5: every job passed except docker-e2e shards 3 and 4, which hit the same inherited failures as on 9ce0f3b:

    • Shard 3: the pattern-p01-dependency-bootstrap docker inspect … No such object race.
    • Shard 4: native-config-build and native-config-down-hooks (also failing on the next base run 39663080763).

    The run on 94e12a9 was in progress when this was written. Shards 1–3 and all other completed jobs passed.

Remaining gaps

  • The Compose-doctor variant is covered by a unit test of its result, not a CLI run, because the private CLI harness does not stub its Docker-era checks.
  • Only the explicit platform field is classified. An image with no platform that publishes only amd64 is still refused later, at native image admission, with the existing image refusal.
  • Only .hack/docker-compose.yml is classified. Compose override files and env-overlay interpolation are not evaluated. An interpolated platform such as ${PLATFORM} is treated as non-native.
  • The check validates only the version and isolatedNetworks parts of the adaptation. Native start still validates the rest of the selection, and that each listed name is an external network, in its existing order. A listed network declared in the legacy external: {name} form passes this check. Native start's adaptation then decides it as before, because replaceNetworks accepts only external: true in the normalized file.
  • Native support for these features stays in HACK-1270. Real-project acceptance has not been run here.

Refs HACK-1291

🤖 Generated with Claude Code

…losed refusals

Native image-project `up` and `restart` now refuse an external Compose
network, a reserved ownership label (com.docker.compose.*, io.hack-local.*,
io.hack.native-config.*), a build outside the import preview's basic form,
or a non-arm64 service platform before branch, env, lifecycle, storage or
runtime work. Each refusal is one value-free E_NATIVE_PROJECT_UNSUPPORTED
line that names the feature and the HACK_RUNTIME_BACKEND=compose selection.

`hack doctor` reports the same features in a `compose backend features`
check, and the native candidate guide lists which Compose features keep a
project on the Compose backend.

Refs HACK-1291

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Oct 21, 2026

Copy link
Copy Markdown

HACK-1291

…se-feature-refusals-b21

# Conflicts:
#	src/commands/doctor.ts
@blacksmith-sh

This comment has been minimized.

…ation

The HACK-1291 Compose-backend check read only the original Compose file, so an
external network that a valid HACK_NATIVE_ADAPTATION selection isolates (the
documented `isolatedNetworks` path) was refused before native start applied
the adaptation. Native up, restart, restart --recover and native doctor now
apply the selection's isolatedNetworks first. The selection is read only when
the file has an external network; an invalid selection refuses with the same
closed E_CONFIG_INVALID line native start gives, before any runtime request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@blacksmith-sh

This comment has been minimized.

… CLI case

Under heavy host load the stub runtime's reply can time out or arrive
unreadable. Each of those lines is only reachable after the Compose-backend
check passed, so the adaptation case accepts all three.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@blacksmith-sh

blacksmith-sh Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Found 1 test failure on Blacksmith runners:

Failure

Test View Logs
native-config-build (125.6s) — Native --profile must succeed/
native-config-build (125.6s) — Native --profile must succeed
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@roodboi
roodboi marked this pull request as ready for review October 22, 2026 00:46
@roodboi
roodboi merged commit 69ec78a into next Oct 22, 2026
27 of 32 checks passed
@roodboi
roodboi deleted the wf/g2-native-compose-feature-refusals-b21 branch October 22, 2026 00:46
roodboi added a commit that referenced this pull request Oct 22, 2026
…l untracked (#369)

## Root cause

Two gaps in the candidate guides (HACK-1315):

1. The candidate install guide told users not to run stable `hack` and `hack-next` against the
same home at once, but it did not cover the stable `hackd` daemon. Stable `hack global up`
(v4.2.0) runs `hack daemon start` whenever `daemon.autoStart` is on, which is the default
(`src/commands/global.ts`, `src/control-plane/sdk/config.ts`). Other stable commands that talk
to `hackd` also auto-start it (`src/daemon/client.ts`). So a user who stopped `hackd` and then
ran `hack global up` to bring up routing had a stable daemon running on the same `~/.hack`
during the Compose candidate run. 2. Nothing in the candidate guides said not to commit
`.hack/.internal/`. A project that tracks it (Omega does) gets a dirty working tree every time a
newer generator rewrites files such as `compose.override.yml`.

## Fix (docs only, no product behavior change)

- `docs/guides/candidate-install.md`: new subsection "Stop stable hackd before a Compose run"
  with the order `hack global up`, then `hack daemon stop`, then check that `pgrep -fl "hack
  daemon start"` prints nothing, and no stable `hack` commands while the candidate run uses the
  same `~/.hack`. New subsection "Keep `.hack/.internal/` out of Git" pointing at the existing
  `hack doctor` generated-files check and `hack doctor --fix`. The shared-state bullet list
  links to both.
- `docs/guides/native-candidate.md` (copied into the bundle as `README.md`): the `hack-v5`
  Compose paragraph now gives the same order and the `.hack/.internal/` rule.
- `docs/guides/v5-dogfooding.md` step 2: links to the new hackd subsection and repeats the
  order.
- `docs/architecture.md`: `.hack/.internal` is machine-local; do not hand-edit **or commit** it.

## Doctor warning follow-up

No new code was needed. A doctor warning for tracked `.hack/.internal/` files already exists:
`src/lib/doctor-generated-files.ts` ("generated files" check, pathspec `.hack/.internal`), and
it also ships in stable 4.2.1. The new guide text points users to it. `src/commands/doctor.ts`
was not touched (in-flight PR #365).

## Tests and evidence

- Bun 1.4.2 (`mise x bun@1.4.2`).
- Local focused: `tests/cli-reference-docs.test.ts`, `tests/prerelease-plan.test.ts` (checks
  `native-candidate.md` anchors), `tests/doctor-generated-files.test.ts`: 66 pass, 0 fail.
- `bun run --cwd packages/cli typecheck` exit 0; `bun run --cwd packages/cli check` exit 0
  (warnings only, none new).
- Relative links and anchors in the 4 changed files: checked with an ad-hoc script, 0 broken.
  The script was proven to flag a bad anchor with a probe file.
- gitleaks 8.30.1 over `origin/next..HEAD`: no leaks.
- Full suite on the M3 at exact head `ea9b4ac8` (Bun 1.4.2): 7554 pass, 126 skip, 1 fail, across
  540 files. The failure was `tests/native-authored-project-recovery.test.ts` "new no-host death
  uses real read-only quiescence facts and retires only its exact bound record" (`Native process
  live-session observation refused (changed)`). Serial reruns with `--timeout 60000` failed
  twice on the M3 and once locally, and each also failed "new no-host recovery retains original
  bindings on last-await". **The same 2 tests fail at bare `next` `ec223cdb` on the M3**, so the
  failure already exists on `next` and is unrelated to this change. This test does not read any
  doc changed here. It is reported as a separate finding.
- No CLI surface or consumer agent guidance change, so `docs:cli-reference` and `generate:agent-
  plugins` do not apply. No unit tests were added, because this is a docs-only change.

## Gaps

- I checked the commands against the source, not on a host with stable 4.2.0 installed: `global
  up` auto-start, `daemon stop`, and the `daemon start --foreground` argv that the `pgrep`
  pattern matches.
- A launchd-managed `hackd` normally exits 0 on SIGTERM, so launchd
  (`KeepAlive.SuccessfulExit=false`) should not restart it. The guide still tells users to re-
  check with `pgrep` before starting the candidate.

Refs HACK-1315

🤖 Generated with [Claude Code](https://claude.com/claude-code)
roodboi added a commit that referenced this pull request Oct 22, 2026
…hase (#375)

## Summary

Fix unit G30: native e2e and storage budgets (HACK-1324, HACK-1320, HACK-1321).

### HACK-1324: storage preparation gets its own named budget

Root cause, confirmed on three hosted first attempts (#365 twice, #370):

- Every storage carrier took its absolute deadline from the start budget
  (`Date.now() + resolveComposeStartupTimeoutMs()` in
  `src/lib/native-compose-command-storage.ts`). `native-config-down-hooks` runs
  with `HACK_COMPOSE_STARTUP_TIMEOUT_MS=15000`, so its first storage carrier had
  15 s for the dependency check, target observation and helper
  create/policy/start. On loaded Blacksmith runners preparation alone takes
  longer, and the scenario failed at 18.3-18.5 s with `storage-target`,
  `storage-helper-*`, `probe-timeout`/`probe-cancel`/`unclassified`.
- In `native-config-build`, about a dozen full storage proofs run in the build
  and pre-spawn guards (`fresh()` in `buildNativeComposeImages`, then
  `runNativeComposeOwnedFileChild`). They consumed the 90 s start deadline, so
  the spawn admission refused (`compose-child` / `private-uncertain`, run
  39676161586) or the outcome was `E_STARTUP_INCOMPLETE` (run 39663510147).

Changes:

- `HACK_NATIVE_STORAGE_TIMEOUT_MS` (`resolveNativeStorageTimeoutMs`, default
  90 s, range 1,000-600,000 ms) bounds each storage carrier. It is never derived
  from the start budget. Default behavior on an unset host is unchanged (the old
  start default was also 90 s).
- `createNativeComposeStartDeadline`: guard storage proofs during a native
  `up`/`restart` are excluded from the start budget; their monotonic elapsed time
  extends the start deadline. Overlaps count once, rejected proofs still count,
  the deadline never moves earlier, and each excluded proof stays bounded by its
  own carrier budget. Build and spawn admission read the live deadline.
- Whole-phase cap on excluded storage time (7daca0d): the total excluded from one
  `up`/`restart` start deadline may not exceed 16 x `HACK_NATIVE_STORAGE_TIMEOUT_MS`
  or 1 hour, whichever is smaller (24 min at the 90 s default; 1 h from S = 225 s
  up). The deadline moves at most that far; a guard proof that would start past
  the cap is refused before it runs; one that ends past it refuses the phase with
  a closed, value-free `NativeStorageTotalBudgetError` (effect reason
  `storage-budget`, stage `guard-storage`). The phase ends within the start
  budget + the cap + at most one in-flight guard proof.
- Closed effect reason `storage-budget`: a refusal that finds its carrier's own
  deadline passed (not cancelled) names the budget; the human summary names
  `HACK_NATIVE_STORAGE_TIMEOUT_MS`.
- Scenario budgets: build hack commands 120 s -> 300 s, down-hooks 90 s -> 240 s.
  Both scenarios now log per-command duration plus closed storage phase timings
  from the passive phase trace, so hosted runs record measured preparation
  durations.

No check, ownership fence, retention or recovery rule changed; per-probe caps (10-15 s) are
unchanged.

### HACK-1320: networks whole-run budget

- Named `NATIVE_NETWORK_FIXTURE_SCENARIO_TIMEOUT` = 90 min. 44 measured hosted
  passes took 2113-3494 s (median ~2650 s); this PR's own run hit 3494 s, 97% of
  the old 60-minute budget. 90 min keeps 1.55x the slowest pass; with cleanup and
  two command bounds the hard bound stays inside the 150-minute shard job.
- Whole-run and cleanup budgets now gate only the start of a command; a started
  command keeps its full 180 s. Run 39431629301 failed because the remaining
  whole-run budget clipped and SIGKILLed an in-flight `down` carrier command,
  leaving a created helper that saved `down --recover` correctly refuses to
  repair.

### HACK-1321: M6 and monotonic group absence

- The unrecorded-command group-absence deadline is now monotonic
  (`performance.now`), named `UNRECORDED_GROUP_BUDGET_MS` (3 s).
- New fake-clock test pins the `probe-budget` refusal on the no-record path.
- Carrier doc lists the fixed 10-15 s probe caps.

### Release note

- Storage carriers no longer follow `HACK_COMPOSE_STARTUP_TIMEOUT_MS` in either
  direction (a short start budget no longer shortens them, a long one no longer
  lengthens them). Hosts that set `HACK_COMPOSE_STARTUP_TIMEOUT_MS` should also
  review `HACK_NATIVE_STORAGE_TIMEOUT_MS`, which now sets both the per-carrier
  bound and the whole-phase storage cap.

## Evidence

### Rework at 7daca0d (verifier gaps)

- Wiring test: `tests/native-compose-start-storage-exclusion.test.ts` runs the
  real source CLI `up` with a 3 s start budget. The Docker stand-in reads the
  CLI's own phase trace (stderr written to a file, so it is on disk before the
  carrier spawns Docker) and delays the carrier dependency check (`image
  inspect`) by 4 s only while a `guard.storage` span is open. No call counting.
  About 43 s on a loaded Mac (the earlier prototype took about 80 s). Mutation:
  with `start.excludeStorage(...)` removed around the guard's `storage.verify`,
  the test fails (`up` exits 1, the workload is never admitted); restored, it
  passes.
- Cap tests in `tests/compose-startup-budget.test.ts` (fake clocks). Each of four
  mutations fails them: no post-proof check, no entry check, no `min` on the
  deadline extension, and the cap error not classified as `storage-budget`.
- Docs: `docs/cli.md`, `docs/reference/native-compose-commands.md` and
  `docs/reference/native-compose-storage-xattr-carrier.md` describe the per-proof
  bound, the phase cap and how the two variables interact.
- Gates: CLI and root typecheck, root `bun run check` (privacy-check ok), biome
  on the changed files (only an existing complexity warning in
  `native-compose-command.ts`), gitleaks 8.30.1 over `origin/next..HEAD` (no leaks).
- M3 full suite at 7daca0d: 7721 pass, 126 skip, 1 fail
  (`repository catalog > the checked-in .hack/tasks.json recipes are all
  reviewed`). It passes locally at the same head and in hosted `test`, and this
  PR does not touch `.hack/` or the task catalog. I could not rerun it on M3 next
  because SSH to M3 failed (local agent signing error).
- Hosted CI at 7daca0d: all checks pass, including all four docker-e2e shards.

### Measured storage durations (hosted shard 4, this branch, 9 attempts)

From the new per-command phase-trace lines:

| Measure | Range across 9 attempts |
| --- | --- |
| Longest single storage proof | 3.6-11.7 s |
| Cold enrollment (`storage.enroll`, first `up`) | 11.7-35.0 s |
| `native-config-build` first `up` | 50-176 s, of which 16 proofs took up to 145 s |

Old behavior on the slowest attempt: the 145 s of guard proofs alone exceed the 90 s start
budget, and the 35 s cold enrollment exceeds the 15 s budget the down-hooks scenario gave every
carrier. New budgets: 90 s per carrier (2.6x the slowest observed carrier work, 7.7x the slowest
proof); build hack commands 300 s (1.7x the slowest `up`); down-hooks hack commands 240 s.

### Hosted CI on this branch (11 attempts across 0af0fb6, 94ce188, cee65eb, 4e2b056)

| Scenario | Passes | Durations |
| --- | --- | --- |
| native-config-build | 11/11 | 116.7-336.9 s |
| native-config-down-hooks | 11/11 | 84.9-181.4 s |
| native-config-networks | 11/11 | 1766-3494 s |

The merged head 4e2b056 (next a25160d merged in) passed both of its full runs (run 39721132414
attempts 1 and 2). Before this change, shard 4 failed on first attempts of #365, #370, #374 and
on next itself (39663080763, 39670689406).

Other attempt failures, all unrelated to these files: `tests/native-authored-project-
start.test.ts` frontend cascade (also seen on 6+ other branches and next), `tests/private-env-
file.test.ts` symlinked `.env` ENOENT, and one runtime-core Rust test
(`rebind::tests::healthy_fixed_selection_is_noop_and_complete_shared_slot_refreshes_together`).

### Gate vs PR shard-4 comparison

Same runner pool and labels (`blacksmith-4vcpu-ubuntu-2404`, runner 2.337.0), identical shard
composition and order, no leftover volumes before `native-config-build` (end-of-job cleanup
volumes are the failed scenarios' own retained fixtures). The failing jobs (including next push
run 39663080763) ran unrelated earlier scenarios 1.3-1.9x slower, so the difference is runner
speed, not composition or state. The logs print no `docker info` storage-driver lines, so that
dimension could not be compared.

### Mutation proofs (each killed by a focused test)

M6 (`groupAbsent` deadline -> plain `refuse()`), M6b (wall-clock deadline), carrier deadline
from the start budget, no start-deadline exclusion, nested exclusion double-count, build
deadline read once, carrier reason ignores budget, budget reported over cancellation, xattr
lifetime unmarked, effect reason ignores mark, summary not naming the variable, networks command
clipped to the whole-run remainder.

### Local and M3 gates

- `bun run typecheck` and `bun run check`: 0 errors at 4e2b056.
- gitleaks 8.30.1 over `origin/next..HEAD`: no leaks.
- M3 full suite at 0af0fb6: 7587 pass, 126 skip, 0 fail.
- M3 full suite at 4e2b056 (host load 25-39): 7681 pass, 126 skip, 12 fail.
  10 of the 12 pass when their files rerun serially; the other 2
  (`tests/native-authored-project-recovery.test.ts` no-host death/recovery)
  fail the same way on plain next a25160d. None touch the changed code.
- M3 is not a qualifying host for the native e2e scenarios; hosted CI is the
  e2e proof.

### Gaps

- The whole-phase cap is proven at unit level only. A command-level cap test
  cannot isolate it, because at S = 1 s the per-carrier budget fires first.
- An in-flight guard proof is not aborted when the cap is reached. It finishes
  within its own per-carrier bound, then the phase refuses.
- The networks scenario still SIGKILLs a command that exceeds its own 180 s
  bound; that genuine hang still leaves storage evidence for explicit recovery.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
roodboi added a commit that referenced this pull request Oct 22, 2026
…ough it, and support adopted projects (#392)

## Summary

Two `hack doctor` fixes.

- **HACK-1316:** doctor no longer follows a symlinked `.hack` or `.hack/.internal`. Before this
  fix, `doctor --fix` deleted a file outside the project.
- **HACK-1311:** doctor now reports an adopted (`hack config adopt`) project correctly, and
  `doctor --fix` completes in one. Before this fix, doctor refused the project as native input,
  the `--fix` plan threw, and `--fix` with tracked generated files failed after it had already
  changed the Git index.

## Rework after review (commit `ff7f11e1`)

The verifier found one blocking defect in the first version (`ffbc4ffd`):

- **Defect.** In an active adopted project whose generated files are tracked by Git (for example
  a committed `.hack.secret.key`), `hack doctor --fix --no-interactive` exited 1 with
  `E_NATIVE_PROJECT_UNSUPPORTED`. It did this after `git rm --cached` had already run, so
  `.hack/.gitignore` was not written and the later fix steps (hackd, Docker networks, Caddy and
  the rest) never ran.
- **Root cause.** `maybeUntrackGeneratedFiles` selected the adopted project, ran `git rm
  --cached`, and then called `ensureHackDirGitignore`. That writer calls
  `assertLegacyProjectDirectory`, which throws `ProjectInputSelectionError("native")` for an
  active adoption (`src/lib/project-input-selection.ts`).
- **Fix.** For an adopted project, the step now changes only the Git index. It still adds a
  committed `.hack.secret.key` to the root `.gitignore`, which is not an adoption input. It
  leaves the adoption's `.hack/.gitignore` unchanged and says so in a closed note: `Removed N
  path(s) from the git index. Left .hack/.gitignore unchanged: an adopted Compose project's
  .hack/ belongs to 'hack config adopt'.` The plan line names the same change. Legacy projects
  behave exactly as before.
- **Docs.** `docs/reference/native-compose-adoption.md` now matches the code: `--fix` leaves
  `.hack/.gitignore` unchanged and adds a committed key to the root `.gitignore`.
- **Test gaps closed.**
  - The `findDoctorLegacyProjectContext` guards on the later `--fix` steps had no regression
  test. The new test runs the whole `--fix` flow with Docker reachable and `--migrate-env-
  config`, so the lifecycle, incomplete-runtime and env-migration steps run against the adopted
  project.
  - The writer's refusal before `mkdir` (HACK-1316) had no test. A new unit test covers it.

## Update: merge with `next` at `5c9cc82e` (commits `0d6aa587`, `51244fc9`)

- `0d6aa587` merges `origin/next` at `5c9cc82e` (#375, #407, #412) with no conflicts. #407
  changes the same `.hack/.gitignore` code in doctor: the repair helper now returns `{ changed,
  readOnly }` and the untrack step can leave a read-only file alone. After the merge, the
  adopted branch of the untrack step still skips the `.hack/.gitignore` writer, and only the
  legacy branch uses #407's read-only handling.
- `51244fc9` is a one-line docs fix. `docs/reference/native-compose-adoption.md` listed every
  check that doctor skips for an adopted project except the `.hack/.gitignore` check. The code
  already skips it.
- **The verifier's repro, rerun on the merged head.** It is the verifier's own scratch test:
  `retainedSourceBindFixture`, a real `git init`, `git add -f .hack.secret.key`, adopt (prepare
  and publish), then `runCli doctor --fix --no-interactive`, with shell `run` mocked. The output
  has no `E_NATIVE_PROJECT_UNSUPPORTED`. `git ls-files` is empty, `.hack/.gitignore` is not
  created, the closed note above is printed, the adoption is still `active`, and the later steps
  run (the global `daemon start` call and the closing `Re-run: hack doctor` note).
- **Why that run still exits 1.** Doctor's exit code comes from the checks that run before
  `--fix`. This is existing behavior and is the same for legacy projects. The verifier's harness
  makes every non-Git `exec` fail, so `docker daemon` is an error, and the committed key is an
  error in the `generated files` and `env key` checks. The PR's `doctor-fix-adopted-project`
  test makes Docker reachable. Its `.hack/.env.state.json` case exits 0. Its key case exits 1
  only because of the committed-key error.
- **Mutations rerun on the merged head**, each killing its test for the intended reason:

| Mutation | Result |
  | --- | --- |
  | Adopted untrack step writes `.hack/.gitignore` again (`ffbc4ffd` behavior) | `doctor-fix-
  adopted-project` 0/2, with `E_NATIVE_PROJECT_UNSUPPORTED` in the output |
  | `findDoctorLegacyProjectContext` no longer skips an adopted project (survived the first
  review) | `doctor-fix-adopted-project` 0/2, with `E_NATIVE_PROJECT_UNSUPPORTED` |
  | Writer's pre-`mkdir` refusal removed (survived the first review) | `compose-env-at-rest`
  15/16; "the writer refuses a symlinked .hack before creating .internal behind it" fails |

## Root cause

### HACK-1316: symlinked `.hack/.internal`

`src/lib/compose-env-at-rest.ts` checked that `.hack/.internal` was a real directory, but only
inside `readReplaceablePrivateFile`, and it never checked `.hack`. When that read refused,
`inspectEntry` called `lstat(path)` to choose between `absent` and `unverifiable`. That `lstat`
followed the `.internal` link, found the outside `compose.env.override.yml`, and classified it
as `unverifiable`. With `includeUnverifiable`, the `--fix` removal then ran `lstat`, `rename`
and `unlink` through the same link, which deleted the outside file. The stale-aside sweep ran
`readdir` on the linked directory before any symlink check, so it could also unlink
`.remove`-shaped files in the outside directory.

### HACK-1311: adopted projects

An active or pending adoption makes `inspectProjectInputsAtRoot` classify the root as `native`.
Doctor's first step, `assertLegacyProjectDiscovery`, therefore threw
`E_NATIVE_PROJECT_UNSUPPORTED`. Every project check and `buildDoctorRemediationPlanLines` call
`findProjectContext`, which throws the same refusal. Every `--fix` step that writes legacy
project files does too.

## Fix

### HACK-1316 (`src/lib/compose-env-at-rest.ts`)

- One no-follow ownership check now covers both `.hack` and `.hack/.internal`. The writer and
  every read, removal and sweep use it.
- A new `refused` state covers these cases:
  - a symlinked or non-directory `.hack` or `.hack/.internal`;
  - a symlinked or non-regular override.

Doctor reports it with a closed, value-free message. A `refused` override is never eligible for
  removal.
- Removal returns before the sweep when the directories are not real. It checks the directory
  identities again before the move and before each aside unlink. The aside listing runs only on
  the held directory.
- The writer refuses a linked `.hack` or `.internal` before `mkdir`, so it creates nothing
  through the link.

### HACK-1311 (`src/commands/doctor.ts`, `src/lib/doctor-adopted-project.ts`)

- Doctor selects an active or pending adoption first, with `selectLegacyComposeAdoptedRoot`, the
  selector that adopted `ps`, `logs` and the other adopted commands use. The legacy-discovery
  refusal runs only when no adoption is found.
- **Checks for an adopted project:**
  - The `project` check names the adopted root. A pending adoption is a warning that asks for
  explicit recovery.
  - The env key, tracked generated files and Compose env at rest checks run against the adopted
  root.
  - Each check that reads adoption-owned inputs is listed as not checked.
  - Agent integrations use the adopted root, and sessions mux uses global settings.
- **`--fix` for an adopted project:**
  - The plan says it leaves the adoption's inputs, retained containers and state alone, and it
  skips env migration.
  - Lifecycle, incomplete-runtime and env-migration repairs treat the project as having no
  legacy project.
  - Untracking generated files changes only the Git index (see the rework above).
- The legacy project-check block moved unchanged into `runProjectChecks`, to stay under the
  complexity limit. The #365 compose backend features check and the #357 env-at-rest adopted
  fallback are unchanged.

There are no CLI surface changes and no agent-guidance changes.

## Tests

| Test | Proves | Mutation that kills it |
| --- | --- | --- |
| `doctor-fix-adopted-project` (new), 2 cases: a committed `.hack.secret.key` and a committed `.hack/.env.state.json` | Real `runCli doctor --fix --no-interactive --migrate-env-config` in an adopted fixture with a real `git init`. Docker is reachable through the mock, and Git runs only on the fixture with no global or system config. Each case checks:<ul><li>no `E_NATIVE_PROJECT` and no canary or key text;</li><li>the exact plan line and step note;</li><li>the closing notes, which prove every later step ran;</li><li>`git ls-files` is empty and the file is unchanged on disk;</li><li>the root `.gitignore` gets the key entry only in the key case, and `.hack/.gitignore` is not created;</li><li>the receipt is unchanged, the adoption is still active, and no stub engine command or project `run` call happens.</li></ul>Exit code: 0 for the env state file. For the key it is 1, because the `env key` check reports the committed key as an error before `--fix` runs. That is existing doctor behavior. | <ul><li>`ffbc4ffd` behavior (always write `.hack/.gitignore`): 0/2 pass, exit 1.</li><li>`findDoctorLegacyProjectContext` no longer skipping adopted projects: 0/2 pass, exit 1.</li><li>Plan line reverted to the legacy wording: 0/2 pass.</li></ul> |
| `compose-env-at-rest`: "the writer refuses a symlinked .hack before creating .internal behind it" (new) | A link to an outside directory without `.internal` gets nothing created behind it, and the refusal is closed. | Writer's pre-`mkdir` refusal removed: fails (this mutation survived before). |
| `doctor-fix-noninteractive`: "doctor --fix leaves a file behind a symlinked .hack/.internal untouched (HACK-1316)" | Real `runCli doctor --fix`: the outside file survives, and the plan and notes do not name it. | `origin/next`'s helper: fails with ENOENT, because the outside file was deleted. |
| `compose-env-at-rest`: symlinked `.hack` and `.hack/.internal` (2 cases) | `refused` state, closed message, no repair planned, no asides removed with a stale-aged clock, writer refusal, outside contents byte-identical. | Old helper: both fail. Directory list without `.hack`: the `.hack` case fails. `inspectEntry` always `unverifiable`: the symlinked-override case fails. |
| `doctor-adopted-project`: `doctor --json` and the `--fix` plan in an adopted project | Closed project message, the checks that run, the not-applicable list, the exact plan, the receipt unchanged, no container start or stop. | Adopted branch of `runProjectChecks` disabled: fails. Preflight forced back to legacy discovery: fails. |

All tests use temporary directories only. They do not touch the real HOME, `~/.hack`, the Docker
socket, the keychain or launchd.

## Evidence

- **Toolchain.** Bun 1.4.2 (`mise x bun@1.4.2`), `bun install --frozen-lockfile`. `bun.lock` is
  unchanged.
- **Local gates on the current head `51244fc9` (M5):**
  - `bun run --cwd packages/cli typecheck`: 0 errors.
  - Root `bun run check`: exit 0, including `privacy-check: ok` (34 existing warnings in other
  files).
  - Biome on the 7 changed source and test files: clean.
  - gitleaks 8.30.1: `gitleaks dir` over a `git log -p origin/next..HEAD` export: no leaks.
- **Focused tests at `51244fc9`, each file run on its own with `--timeout 120000` (the M5 was
  under heavy load), all passing:**
  - `compose-env-at-rest` 16, `doctor-fix-noninteractive` 10, `doctor-adopted-project` 2,
  `doctor-fix-adopted-project` 2
  - `native-compose-backend-features` 13, `native-compose-backend-features-cli` 15, `compose-
  env-at-rest-cli` 9
  - `doctor-command` 28, `doctor-json` 3, `doctor-generated-files` 11, `doctor-env-key` 31,
  `doctor-utils` 8, `doctor-worktree` 6
  - From #407 and #412, which touch the same code: `hack-gitignore` 14, `project-env-config` 58,
  `project-env-hydrate` 18, `compose-env-shared-references` 6
- The five doctor `--fix` and at-rest files also pass together in one process (41/41), which
  checks that the scoped module mocks do not interfere.
- Under the same load, `doctor-fix-adopted-project` with Bun's default 5 s timeout failed in a
  fixture hook. It passes with the longer timeout. This is load, not a code change.
- **Full suite on the M3: not run.** The runner for `91dae070` waited about 6.5 hours and never
  got the shared CI lock. I stopped it and queued a runner for `51244fc9`, which waited about 50
  minutes. During that time the lock directory had existed since 12:37 with no suite process
  running on the M3, several other units were also waiting, and free space was about 57.6 GiB,
  under the 58 GiB floor. I did not touch the lock, since I could not prove who owned it. I
  stopped my runner. It created no worktree or ref.
- **Hosted CI:** all 16 checks pass on the current head `51244fc9` (run 39839971608): the full-
  suite `test` job (39 min), all four `docker-e2e` shards, candidate core, config compiler,
  runtime state models, and secret scan. Earlier heads `0a94d881`, `a2470049` and `91dae070`
  also passed.

## Remaining gaps

- **No M3 full suite.** It did not run because the shared lock was blocked and free space was
  under the floor (see Evidence). The hosted full-suite `test` job ran at `51244fc9` on ubuntu,
  not on macOS.
- **Symlinked `.hack` now refused by the writer.** A Compose command that renders the env
  override now refuses a symlinked `.hack`, not only a symlinked `.internal`. This matches the
  existing `.hack/.env` rule. A project that relied on writing through a linked `.hack` will see
  a closed refusal.
- **Small race window remains.** Directory identities are checked again before each move and
  unlink, not held as open descriptors. A same-user swap in that window remains, as already
  documented for the move-aside window.
- **Other generated files can be committed again.** For an adopted project, `--fix` does not add
  the untracked generated files under `.hack/` to any ignore file. Only the key gets a root
  `.gitignore` entry. The note asks the user to keep them out of later commits.
- **`--fix` with a committed key exits 1.** The exit code reflects the checks that ran before
  the fix, for legacy and adopted projects alike. The fix itself completes and removes the key
  from the Git index, which is what the tracked-key checks read.
- **Pending adoption not tested.** The pending-adoption warning has no test: the fixture has no
  cheap pending state.
- **Native backend skips adopted project checks.** With `HACK_RUNTIME_BACKEND=native`, native
  doctor still skips project checks for an adopted project. This is unchanged.

Refs HACK-1316 Refs HACK-1311

🤖 Generated with [Claude Code](https://claude.com/claude-code)
roodboi added a commit that referenced this pull request Oct 24, 2026
…ls, null platform and external volumes (#404)

## Root cause

The native Compose-backend classifier from #365 (`src/lib/native-compose-backend-features.ts`,
used by native `hack up`/`hack restart` and `hack doctor`) had four gaps:

1. **The reserved label set was written by hand and was incomplete.** It listed
`com.docker.compose.`, `io.hack-local.` and `io.hack.native-config.`. It missed
`io.hack.storage-witness.*` (storage witness carriers) and `io.hack.native-file.*` (VM file
ownership). A service labeled `io.hack.storage-witness.owner: x` got the general native
configuration refusal, not the closed Compose-backend refusal. No single source listed Hack's
ownership label namespaces: the keys were string literals in about 16 TypeScript files and in
runtime-core. 2. **`platform: null` was classified as a foreign platform.** Compose and native
review both treat a null platform as unset (runtime-core's `optional_text` filters null). 3.
**External volumes were not classified.** Runtime-core refuses them as `external_volume`, but
the classifier did not report them. 4. **`checkComposeBackendFeatures` was placed between the
HACK-1127 JSDoc and `checkProjectEnvKey`**, so that JSDoc was attached to the wrong function.

## Fix

- New `src/lib/hack-ownership-labels.ts` exports `HACK_OWNERSHIP_LABEL_NAMESPACES` (`io.hack-
  local.`, `io.hack.native-config.`, `io.hack.native-file.`, `io.hack.storage-witness.`). This
  list is now the single source of truth. `RESERVED_OWNERSHIP_LABEL_PREFIXES` is
  `com.docker.compose.` plus that list, and the refusal and doctor wording are built from it.
- A parity test (`tests/hack-ownership-labels.test.ts`) scans `src/**/*.ts`,
  `packages/*/src/**/*.{ts,rs}` and `packages/*/tests/**/*.rs`. The scan is read-only. It checks
  two things: every `io.hack*` label key falls in a declared namespace, and every declared
  namespace is used somewhere. The two files that only restate the list are left out of the
  scan.
- An absent or `null` `platform` is unset. Any other non-native value, including `""`, numbers
  and booleans, is still `foreign_platform`.
- New closed feature `external_volume` (ordered after `external_network`). It covers
  `volumes.<v>.external: true` and the legacy `external: {name: ...}` form. It appears in the
  refusal, the doctor text and JSON (`composeBackend.features`), and the docs table.
- Moved `checkComposeBackendFeatures` above the HACK-1127 JSDoc, so `checkProjectEnvKey` has its
  JSDoc again.
- Documented that only `.hack/docker-compose.yml` and the explicit `platform` field are
  classified. Overrides, `extends`, `include` and `DOCKER_DEFAULT_PLATFORM` are not followed,
  which matches native start, because `prepareNativeProjectInput` reads only that one file.
  Extending the classifier to them is not cheap: native start does not merge them either, and
  the native runtime's own review reports `extends` and `include` as unsupported fields.

The CLI interface did not change, so `docs/reference/cli.md` is not regenerated. Consumer agent
guidance and `package.json` scripts did not change.

## Tests (each new behavior was checked by a mutation)

| Mutation | Tests that failed |
| --- | --- |
| Remove `io.hack.storage-witness.` from `HACK_OWNERSHIP_LABEL_NAMESPACES` | parity scan, reserved-namespace unit test, CLI storage-witness refusal, CLI reserved-label and combined-feature refusals (5) |
| Add a stale `io.hack.unused.` namespace | "every declared ownership namespace is written or verified somewhere" |
| Treat `platform: null` as foreign again | unit platform test, CLI null-platform test (2) |
| Disable `external_volume` detection | unit external-volume and order tests, CLI external-volume refusal, CLI combined refusal (4) |
| Accept only `external: true` (drop the legacy `{name}` form) | unit external-volume test |

New CLI tests use the real source CLI with a stub runtime. They show the following:
- `io.hack.storage-witness.owner` refuses native `up` with the closed
  `E_NATIVE_PROJECT_UNSUPPORTED` line. The runtime received no requests and no files were
  written.
- An external volume is refused the same way.
- A null platform passes the check and reaches `runtime up`, so native review admits it.
- `extends` and `include` are not followed into another file.
- Native doctor JSON reports `external_volume` and leaves out the volume name.

## Evidence

- Bun 1.4.2 (`mise x bun@1.4.2`), `bun install --frozen-lockfile`. `bun.lock` is unchanged.
- Local (this Mac): `bun run --cwd packages/cli typecheck` passed. Root `bun run check` passed,
  including `privacy-check: ok` (only existing warnings). Biome on the changed files is clean.
- Focused tests, local: `tests/hack-ownership-labels.test.ts`, `tests/native-compose-backend-
  features.test.ts` and `tests/native-compose-backend-features-cli.test.ts` all pass. So do
  `tests/doctor-command.test.ts`, `tests/doctor-json.test.ts`, `tests/doctor-env-key.test.ts`
  and `tests/cli-reference-docs.test.ts`.
- gitleaks 8.30.1 over `origin/next..HEAD` (a `git log -p` export scanned with `gitleaks dir`):
  no leaks.
- Merged the current `next` (d0c8184) into the branch as d4e4500 (fast-forward push, no
  force). The merge was clean. At d4e4500: CLI typecheck passed, root `bun run check` passed
  (`privacy-check: ok`), Biome on the changed files is clean, and the 98 focused tests above
  pass. gitleaks 8.30.1 over `origin/next..HEAD` found no leaks.
- Mutations re-run at d4e4500: dropping `io.hack.storage-witness.` fails 2 unit tests; treating
  `null` as foreign fails 1; disabling external-volume detection fails 2; the unmutated baseline
  passes 17/17.
- Full suite on the M3 at d4e4500: pending. It is queued behind another unit's long loaded-
  repetition run on the shared M3 lock. This description will be updated with the result.

## Gaps

- The TypeScript and Rust writers still spell their label keys inline. They do not import
  `HACK_OWNERSHIP_LABEL_NAMESPACES`. The parity scan is what enforces the list: a new `io.hack*`
  label namespace anywhere in those trees fails the test until it is declared. Swift and other
  packages are not scanned, and none of them write `io.hack*` labels today.
- The doc statement that the runtime review refuses `extends` and `include` comes from runtime-
  core's `keys()` allow-list (`unsupported_field`). The stub-runtime CLI test cannot exercise
  the real runtime review.
- No native host or VM run was done. The CLI tests stop at the stub runtime.

Refs HACK-1337

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant