[roodboi]

Root cause

On the Compose backend, resolveModernComposeEnvOverrides in src/commands/project.ts rendered every resolved env value into .hack/.internal/compose.env.override.yml. That included values decrypted from secure: v1: entries. It wrote the file with writeTextFileIfChanged (Bun.write, which uses the umask mode and follows a symlink) and never removed it. Up, down, a failed up and an interrupted up all left the plaintext on disk. A rerun on a real project saw this: a 71 KB, 0600 file covering 14 services was still there after down.

The first version of this PR moved the values out of the file. Review found that this broke legacy Compose adoption. Each container's com.docker.compose.config-hash label is computed by Compose with the real HACK_ENV_* values. The adoption probe (inspectLegacyComposeRuntimeConfig) recomputed the hash from the files with no values, so every reference resolved blank. The hashes differed and adoption refused every stack with managed env that this version started. The docker e2e adoption harness also started its original stack with raw docker compose and no values. That failed docker-e2e shards 3/4 and 4/4 (run 39293621453).

Rework 2 (head 879e0b56)

A verifier checked 3970f175 and reported six defects. Commits 5695f824, bebeb89c and 7345de3f already claimed fixes for most of them. This rework checked each claim with a mutation, merged current next, and closed what was still open.

Verifier defect	Status at 879e0b56	Evidence
1. Adoption refuses every stack this version starts (probe hashes ${HACK_ENV_*} as blank)	Fixed in 5695f824	Without privateInterpolationEnv in the probe env, the HACK-1273 adoption test fails. With the saved-hash path disabled, it also fails.
2. hack down skips the scrub when the down-before hook fails or Compose down throws	Fixed in bebeb89c: the scrub runs before hooks, the remote target and Compose down	With the scrub moved back to after the finally block, "down … even when its down-before hook fails" fails.
3. TOCTOU: inspect, then unlink by path	Fixed in bebeb89c (move aside, classify the moved entry, link back without overwriting)	With a plain unlink by path, "removal keeps a value-free override that replaced the inspected literal file" fails. The remaining small window is now documented (see Remaining gaps).
4a. doctor --fix removal untested	Covered by doctor-fix-noninteractive (from bebeb89c)	With the --fix step removed, that test fails.
4b. No-override scrubs untested	2 of 3 were covered. The third (legacy hack.env.json path where no service gets an override) was still untested. This rework adds a test for it.	With each of the three scrub calls removed in turn, a different test fails.
5. CLI test PATH includes the host PATH	Fixed: PATH is the stub bin plus /usr/bin:/bin only	Code reading
6. Docs overstate or omit points	Fixed in bebeb89c. This rework also states the move-aside window and the aside left by a crash, and describes hack down by what it runs before	docs/env.md, threat model row and gap 4

New in this rework:

• Merge with next (cb2d6226). fix(secrets): converge concurrent key provisioning and add sentinel scan controls #349 had made the literal override private (0600, atomic, writePrivateEnvFileIfChanged). This branch keeps its reference writer instead. The resolution drops the unused import, folds fix(secrets): converge concurrent key provisioning and add sentinel scan controls #349's threat-model row into the HACK-1273 row, and rewrites open gap 4 to cover .hack/.env and the Compose client environment.
• Sentinel scan (c33c3c97). tests/env-secret-sentinel-scan.test.ts from fix(secrets): converge concurrent key provisioning and add sentinel scan controls #349 still allowed plaintext in compose.env.override.yml. The file loses that exception. The scan now requires the override to exist with ${HACK_ENV_ references, to be 0600 and git-ignored, and requires the values to reach the docker compose client env. With next's src/commands/project.ts, the scan fails.
• Legacy no-override scrub test and stale-aside sweep test (879e0b56). The stale-aside sweep (removeStaleAsides) had no test. It now takes an injectable clock (a test seam, no authority). The test proves that a fresh aside is kept, a stale one is removed, and a directory or a non-matching name is left alone. If the sweep becomes a no-op, the test fails. It also fails if the sweep ignores the aside's age.

Fix

• The override holds only Compose interpolation references, such as SECRET_TOKEN: "${HACK_ENV_<12-hex digest of service+key>_SECRET_TOKEN}". The values reach docker compose through its process environment. No Compose command writes a value to disk, so a failed, interrupted or killed up leaves nothing behind (renderComposeEnvReferenceOverride in src/lib/compose-managed-env.ts). The path stays the same because Compose records it in each container's config_files label.
• writeComposeEnvOverride (src/lib/compose-env-at-rest.ts) writes the file with replacePrivateFileAtomic. The file gets mode 0600 and a symlink at the path is refused. Text that holds a literal value is refused before anything is written.
• Adoption config hash. The projection already resolves the managed values at preview and preparation. It now also returns the reference values as a non-enumerable, never-serialized composeEnv. inspectLegacyComposeRuntimeConfig passes them only to the Compose child environment of its config --hash probe (privateInterpolationEnv on createNativeComposeProbe). A saved-generation recheck runs without a key or values by design. When the bound files include the env override, that recheck compares each container's engine label with the hash proven at preparation (meta.runtimeConfig). The Compose copy, the generated override bytes and the managed env revision stay pinned by the existing saved owners. A recreated container therefore still refuses.
• e2e harness. bootstrapOriginal now starts the original stack with the same Compose process env that Hack supplies (managedAdoptionFixtureComposeEnv). writeGenerated also checks that every reference value matches what the writer returned.
• Cleanup of files written by earlier versions:
  • The next Compose command that renders the override replaces the file.
  • A Compose command that renders no override removes the file, but only if it holds literal values.
  • hack down now removes the file first, before hooks, remote dispatch or Compose. A failed down-before hook or a remote target no longer keeps it.
  • hack doctor reports the file by counts only. hack doctor --fix removes it.
• Race-safe removal. removePlaintextComposeEnvOverride moves the entry aside with one atomic rename, then classifies the entry it moved. If a concurrent Compose command had already replaced the file with the reference form, that file is linked back without overwriting anything (a newer file wins). A value-free file is never deleted, but the path is absent between the move and the link-back. A crash between the move and the unlink leaves an aside file. A later removal deletes it once it is more than 60 s old (the clock can be injected for tests).
• Behavior change: a value that contains $ now reaches the container exactly as written.

Tests

The CLI tests use privateCliEnv and a stub docker. Their PATH is now the stub plus /usr/bin:/bin, so doctor --json no longer runs the host's mutagen or brew. No real HOME, Docker socket or keychain is used. Every value is a synthetic canary.

New in this rework:

Test	Proves	Mutation that makes it fail
native-compose-adoption-generation "v3 value-free env override binds the engine config hash computed with the values"	The stub engine hashes every ${NAME} reference with its value from the Compose env, as Compose does. The label is computed with the values and differs from the hash without them. Preview passes, prepare saves the label hash, and a saved withLease passes without a key. A changed label refuses. No value or HACK_ENV_ name reaches the manifest, the report or the docker argv.	Dropping the values at preview, or at prepare; dropping saved from the recheck; ignoring the label in saved mode (4 separate mutations, each fails)
compose-env-at-rest-cli "down … even when its down-before hook fails"	A failed hook (exit 17) still removes the literal file	The scrub moved back to after the hooks (the previous placement)
compose-env-at-rest-cli "up that renders no override …" (env config without values; no env config; legacy contract, no value for these services, new in rework 2)	All three no-override paths remove a literal file: the modern null override, the legacy empty contract, and the legacy contract whose vars apply to no running service	Removing each of the three scrub calls (each fails exactly one test)
doctor-fix-noninteractive "doctor --fix removes a literal Compose env override and keeps a reference-only one"	--fix removes the literal file, keeps the reference file and prints no value	Removing the --fix step
compose-env-at-rest "removal keeps a value-free override that replaced the inspected literal file"	A concurrent rewrite between inspection and removal is kept, with mode 0600 and no aside left	Unlinking by path without the move-aside (the 3970f17 behavior)
compose-env-at-rest "removal sweeps an aside left by a crashed removal once it is stale" (rework 2)	A fresh aside is kept, a stale one is removed, and a directory or a non-matching name is left alone	Making the sweep a no-op; ignoring the aside's age
env-secret-sentinel-scan (rework 2, tightened from #349)	No sentinel value exists in compose.env.override.yml. The file exists with ${HACK_ENV_ references, mode 0600, git-ignored. The values reach the docker compose client env	Running it with next's src/commands/project.ts, which writes the literal file

Earlier tests are kept: up/down delivery through the env with only references at rest, a failed up, an up interrupted by SIGTERM, doctor detection by counts, and the renderer, classifier and writer unit tests.

Local gates on 879e0b56 (macOS, shared host with load average 13 to 50):

• bun run --cwd packages/cli typecheck: exit 0. bun run typecheck: exit 0.
• bun run --cwd packages/cli check: exit 0, 0 errors, 34 warnings (the same count as next).
• Biome on the changed files: clean.
• Each file run in its own bun process, all green: compose-env-at-rest 10, compose-env-at-rest-cli 9, doctor-fix-noninteractive 4, tests/cli-spawn-env-isolation.test.ts 7, env-secret-sentinel-scan 6, compose-env-staleness 11, compose-managed-env 2, doctor-command 28, doctor-json 3, doctor-utils 8, native-compose-adoption-projection 17, native-compose-adoption-managed-fixture 3, native-compose-adoption-local-fixture 3, native-compose-adoption-local 42, native-compose-ownership 65, project-exec 5, project-run 10, project-up-startup-state 67, run-exec-branch-default 4, secret-key-provisioning-convergence 2.
• native-compose-adoption-generation: the six env-related cases (HACK-1273, the v3 saved, v3 drift and managed cases) pass. They were run with the file's 20 s per-test bound raised locally to 180 s, and that change was not committed. At the default bound on this host, every case timed out, including on unchanged cases, because another agent was running the same file at the same time and load average reached 40 to 50. The full file was not run green locally. CI's test job is the evidence for the whole file.
• gitleaks is not installed on this host. CI secret-scan covers it.

Hosted CI:

• On 7345de3f (run 39310729568): test passed. docker-e2e shards 1/4 to 3/4 passed. Shard 4/4 failed only in native-config-process-policy ("Native --profile must succeed", a live-endpoint-membership topology refusal on the native backend). In the same shard, native-compose-adoption-local-worktrees passed.
• On 879e0b56 (run 39439748941), at the time of writing: docker-e2e shards 2/4, 3/4 and 4/4 passed, including native-config-process-policy in shard 4/4. secret-scan, runtime-images, linux-process-lifetime, Runtime state models, Candidate core (ubuntu-latest) and three of the four Config compiler jobs passed. test, shard 1/4, Candidate core (macos-latest) and Config compiler (macos-15-intel) were still pending.

Remaining gaps

• Docker e2e was run in CI only. Shard 4/4 on 7345de3f failed in native-config-process-policy. That scenario uses the native backend, not the Compose env override, and the same shard passed on bebeb89c. The exact-head gate is the CI run on 879e0b56, which includes the merge with next. I did not dispatch any workflow.
• Transient absence during a raced removal. When a removal moves aside a file that a concurrent Hack command has just rewritten value-free, the path is absent until the link-back. A Compose process that opens it in that moment fails and must be retried. POSIX has no "unlink if still this inode", so the design accepts this window. It is documented in docs/env.md and threat-model gap 4.
• Aside left by a crash. If Hack is killed between the move and the unlink, .hack/.internal/.compose.env.override.yml.<uuid>.remove can hold literal values. It is swept only by a later removal (a no-override Compose command, hack down or hack doctor --fix) once it is more than 60 s old. hack doctor does not report it, and an up that renders the override does not sweep it. This is documented.
• Adoption test budget. The HACK-1273 adoption case takes about 16 s alone at load average 14, and the file's per-test bound is 20 s. Like the existing v3 cases, it can time out on a saturated host.
• Compose client env size. The docker compose client process carries one HACK_ENV_* copy of each value per service, so its environment grows with services × keys. On macOS the total limit is about 1 MiB. On Linux, a single NAME=value string over 128 KiB (MAX_ARG_STRLEN) now fails with E2BIG, where the old file path worked. This is documented. A shared name per key for globally inherited values is not done. serviceEnv already merges global values into every service, so sharing names would need scope metadata in the renderer and the adoption projection.
• Direct docker compose use. A docker compose -f .hack/.internal/compose.env.override.yml … run outside Hack gets blank values with only per-variable warnings. This is documented in docs/env.md.
• Saved adoption recheck. The recheck now trusts the preparation-time config hash plus the pinned file bytes and env revision. It no longer recomputes the hash with Compose. A change in Compose's own hashing would therefore not be noticed on that path.
• Older plaintext elsewhere. Plaintext that an earlier version wrote can survive in filesystem snapshots or backups. Rotate the affected secrets if that exposure matters. Running containers still expose their env through docker inspect (gap 9).
• No real-engine check here. This branch was not run on the real-project host or on a real local Docker engine. The rerun evidence was read without changes (<private-workdir>/hack1124-next5-rerun-20261019-1204/). It confirms the original defect on published next.5: cleanup-project.txt lists a 71,130-byte, 0600 compose.env.override.yml after down. That file is no longer at <project>/.hack/.internal/ (checked with stat only; no content was read).

Refs HACK-1273

Rework (independent verification follow-up)

Head ddb6efd0 on top of 879e0b56: bfccb534 (fix), 300beace (test fixture), and two merges of next (375e17cc up to 8c790004, which resolved the compose-managed-env import conflict in src/commands/project.ts; ddb6efd0 up to a388ce27, which auto-merged native-compose-ownership.ts and kept privateInterpolationEnv). Nothing was rebased or force-pushed.

Defect 1 (medium): adopted generations bound to a literal override

Choice: refuse and guard; no in-place migration. A saved adoption pins the override's SHA-256, inode, size, mode, mtime and ctime (generatedProof), and prepare refuses while an adoption is active (E_LEGACY_ADOPTION_BUSY). Migrating in place would mean either rewriting a pinned file, which breaks the proof, or adding a new manifest-rewrite path that weakens it. The existing proof-preserving route already migrates: rollback, then hack up (which writes references; the adoption already refuses values containing $, so the config hash and the container IDs do not change), then adopt again.

• hack config adopt and hack config adopt --dry-run now refuse a literal override before any store, probe or engine call. The message tells the user to run hack up first. The preview reports the field code generated_env_override_holds_values. Recovery and rollback still act on existing generations.
• removePlaintextComposeEnvOverride, which backs doctor --fix, down and the no-override scrubs, keeps the file while the adoption state is prepared, active or pending. It also keeps the file when that state cannot be read safely. It returns adoptionBound: true, and the scrub warns without values.
• The doctor "compose env at rest" check now reaches an adopted project. Before, ordinary discovery refused the project as native, so the check errored. It reports the pinned file with the next step: hack down, hack config adopt --rollback, hack up, then hack config adopt --stop. --fix keeps the file and prints the same step.
• Tests cover both directions. tests/native-compose-adoption-generation.test.ts uses the verifier's repro with a literal-form prepare and publish that model an earlier version. In it, removal returns adoptionBound: true and the inode is unchanged, the doctor check warns with the rollback step, and loadActive() still loads. After rollback, the same removal deletes the file. The CLI test checks that config adopt with [], --dry-run, --stop and --dry-run --json refuses with no state root and no engine command. tests/compose-env-at-rest.test.ts checks that an unreadable adoption state keeps the file and that removing that state frees it.

Defect 2 (low): HACK_ENV_* values reached non-Compose docker children

withoutComposeEnvReferences (src/lib/compose-managed-env.ts) now strips HACK_ENV_<12 hex>[_KEY] names before withDependencyStartupProgress spawns docker inspect/logs/events, and before canReuseRunningDependencyCache spawns docker inspect. docker compose invocations keep the full env. The new tests/compose-env-reference-least-privilege.test.ts records each stub docker child's environment and checks that no HACK_ENV_* key or canary is present while GLOBAL_FLAG is. The wording in docs/security/env-secrets.md, threat-model row 78 and docs/env.md now says this.

Findings

• Leftover asides: hack doctor counts .compose.env.override.yml.<uuid>.remove files and warns without values. --fix deletes stale ones, older than 60 s. A Compose command that renders the override also sweeps stale ones, after its symlink checks, on a best-effort basis. The docs and threat-model gap 4 are updated.
• Migration note: docs/env.md has a new "Upgrading: managed values that contain $" subsection. See the release note below.
• exec warning: resolveExecTargetReady now passes the Compose env to compose ps. tests/project-exec-command.test.ts asserts that the ps env equals the exec env.

Release / migration note

Managed values that contain $ change meaning. Earlier versions wrote values literally into the override, so Compose interpolated $VAR, ${VAR} and $$. Values now reach containers exactly as stored. After upgrading, the affected services' containers are recreated on the next hack up. A value that relied on interpolation must store the expanded value. A value escaped as $$ must store a single $. Separately, a project with an active hack config adopt generation made by an earlier version keeps its literal override until it is rolled back, run through hack up and adopted again.

Mutation proof (each fix reverted alone; the named test fails, then passes when restored)

Fix reverted	Test that failed
Adoption guard in removePlaintextComposeEnvOverride	unit "an override that an adoption state may pin…" and the real-adoption case "…keeps it loadable until rollback frees it"
Preview literal refusal	"v3 saved adoption consumes…" and "CLI adoption and its preview refuse a literal env override…"
config adopt apply literal refusal	"CLI adoption and its preview refuse a literal env override…"
Doctor adopted-root fallback	the real-adoption case (doctor check assertion)
Strip in withDependencyStartupProgress	"dependency progress observers get the Compose env without…"
Strip in canReuseRunningDependencyCache	"the dependency cache reuse probe gets the Compose env without…"
env on the exec readiness compose ps	"exec reaches the running service…" (psEnvs assertion)
Stale-aside sweep in the writer	"a rendering write sweeps stale removal asides…"
Aside counting in the doctor report	"doctor reports leftover removal asides by count…"

Gates on ddb6efd0 (macOS, shared host, load average 13 to 130 during the runs)

• bun run --cwd packages/cli typecheck: exit 0. bun run --cwd packages/cli check: 0 errors, 34 warnings (the same as next). Biome on the changed files: clean.
• Focused files, each in its own bun process, all green: compose-env-at-rest 13, compose-env-at-rest-cli 9, compose-env-reference-least-privilege 2, project-exec-command 5, doctor-fix-noninteractive 4, env-secret-sentinel-scan 6, cli-spawn-env-isolation 7, compose-managed-env 2, native-compose-adoption-projection 17, project-run-command 10, project-up-startup-state 67, dependency-startup-progress 6, native-compose-adoption-publication-diagnostics 6, doctor-command 28, native-compose-ownership 67.
• native-compose-adoption-generation, with the 20 s bound raised locally (not committed): 110 of 114 pass. The env-related subset passes 10 of 10 (all HACK-1273 cases, the v3 cases, v4 typed locals, CLI backend refusal and dry-run). The 4 failures are v7 start/restart/source-drift and v5 retained dispatch. Each timed out at its own explicit 30 s bound. They use neither the env override nor the preview.
• Full bun run test on bfccb534 (before the merges): 7045 pass, 291 fail, mostly cascades after timeouts under load. One was real: the v4 typed-locals fixture adopted over a literal override, which the new preview refusal now rejects. 300beace fixed it by switching that fixture to the reference form. Rerun serially with --timeout 60000: 50 of the 57 other affected files pass. The remaining 7 native files (authored-project-recovery, authored-project-start, compose-down-command, retained-storage-command, storage-activation-command, storage-fence-cost, warm-up-command) fail the same way on unchanged origin/next (8c790004) on this host, so they are not caused by this PR.
• gitleaks 8.30.1 (origin/next..HEAD, 9 commits): no leaks.

Remaining gaps

• The library LegacyComposeAdoptionProjection.resolve still matches the literal form. Only the command boundary (config-adopt.ts) and the preview refuse it, and the tests use the literal match to model an earlier version's generation. A generation that an earlier version prepared but did not publish can still be published after upgrading.
• buildDoctorRemediationPlanLines, like most doctor project checks, still throws in an adopted project because findProjectContext refuses native inputs. Only the compose-env check and its fix step have the adopted-root fallback. This behavior predates this PR.
• The adoption regression cases take 20 to 50 s each on this host under load, so they can hit the file's 20 s per-test bound, as the existing v3 cases do. They were verified locally with the bound raised, and that change was not committed.

🤖 Generated with Claude Code

[linear-code]

HACK-1273