Skip to content

feat(env): idempotent batch hydration from stdin or file - #351

Merged
roodboi merged 16 commits into
nextfrom
wf/f8-hydrate-b16
Oct 22, 2026
Merged

roodboi merged 16 commits into
nextfrom
wf/f8-hydrate-b16

Conversation

@roodboi

@roodboi roodboi commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Linear: HACK-1158 (WU27-HYDRATE). Stacked on #340 (wf/f7-stable-ciphertext-b16), which provides semantic ciphertext reuse and the rename-committed layer write. Review and merge #340 first.

Final behavior

hack env hydrate imports many values into the machine-local env layer in one commit:

<secret-source> | hack env hydrate --secret
hack env hydrate --secret --file .secrets.env
hack env hydrate --secret --file secrets.json --format json --env qa --service api
  • Input never in argv. Entries come from piped stdin (a TTY is refused before reading) or --file (regular files and pipes such as <(producer)). Input is bounded at 1 MiB. The env format is strict (KEY=value, comments, export, 'literal', "escaped"). --format json takes one object of non-empty strings. Repeated keys are refused in both formats; JSON input is not resolved last-wins. One malformed entry refuses the whole input. Messages name at most a line number.
  • Only the machine-local layer is written (hack.env.local.yaml or hack.env.<overlay>.local.yaml). Tracked defaults are untouched. A git-tracked local layer is refused. Inside a git checkout where git cannot say whether the layer is tracked (exit 128 from a safe.directory refusal or a damaged index, or git missing), hydration also refuses, so the tracked check fails closed. Outside any checkout it proceeds.
  • Idempotent. An entry that already holds its value is kept: a secret keeps its ciphertext (F7's semantic reuse) and a plain scalar keeps its YAML type. When nothing changes, the file is not rewritten, so hand edits and comments survive and Git shows no diff. A changed value gets new ciphertext with a fresh IV. A repeat must use the same flags: a plain import over a key the layer stores as a secret is refused (secret_downgrade) rather than silently replacing ciphertext with plaintext.
  • Local overrides preserved. Local entries outside the input, other scopes, and removal markers for other keys are kept. A key in the input replaces that key's local entry.
  • All-or-nothing. The whole layer is published by one rename (F7's atomic write). An interrupted import leaves every key at its previous value.
  • Refusals without partial state (closed, value-free, ClosedCliError):
    • tracked_layer: git tracks the target local layer.
    • tracking_unknown: inside a git checkout, git could not report whether the layer is tracked.
    • unreadable_layer: the local layer exists but is not a valid env file (invalid YAML, wrong shape or version). It is never replaced.
    • wrong_key: a secret already in the layer does not decrypt under the reader key, or no key exists. The key file is checked before HACK_ENV_SECRET_KEY, as readers do. No key is minted.
    • key_env_only: --secret with only HACK_ENV_SECRET_KEY and no key file, because minting a file would replace that key for every layer.
    • secret_downgrade: a plain import of a key stored as a secret in the target scope.
    • locked / stale_lock: hydrations are serialized by .hack/.internal/env-hydrate.lock (exclusive create, holds the PID). A lock from an exited process is reported with its path, never taken over.
    • concurrent_write: compare-and-swap. The layer must still equal the bytes read before staging and again after fsync, just before the rename.
  • Values never printed. Text and --json output list only added, updated and unchanged keys.
  • Cross-command hardening. Unexpected arguments now reports a count instead of echoing the arguments, so hack env hydrate API_TOKEN=value cannot print the value. This applies to every command.

Docs: docs/env.md gets a hydration section that lists every refusal, plus a matrix row. docs/reference/cli.md is regenerated. Pattern card HACK-P08 lists hack env hydrate --file <path> --secret and its remaining gap, re-rendered into plugins/hack/skills/hack-cli/references/patterns.md.

Rework after verification (head fba3e87c)

  • secret-scan (blocker): 57a82c2, pushed separately, builds the synthetic HACK_ENV_SECRET_KEY at runtime and adds the fingerprint of the old line in 06f2623 to .gitleaksignore. CI scans the PR's full history, so the runtime change alone would not have passed. fba3e87 moves that test onto a shared runtime-built key helper.
  • Tests now pin the unreadable_layer refusal (invalid YAML, a non-mapping scope, a wrong version, in secret and plain mode, bytes unchanged) and the reader-key priority (key file plus a different HACK_ENV_SECRET_KEY: plain and secret hydration succeed, and readers open every entry).
  • Fail-closed tracked check: new tracking_unknown refusal, described above.
  • Secret downgrade: new secret_downgrade refusal, described above.
  • JSON duplicate keys: refused with duplicate_key, including escaped spellings such as API_TOKEN. Nested keys and quotes, commas or brackets inside values are not miscounted.
  • Docs: the refusal list in docs/env.md is complete, including unreadable_layer and stale_lock.

Validation (local, macOS host, head fba3e87c)

  • tests/project-env-hydrate.test.ts, tests/env-hydrate-input.test.ts, tests/cli-command.test.ts, tests/integration-ownership.test.ts, tests/cli-spawn-env-isolation.test.ts: 48 pass, 0 fail.
  • Broader set of 16 files (the above, plus agent-instruction-source, cli-reference-docs, env-command-matrix, env-key-argument-redaction-cli, env-value-stdin(-cli), pattern-catalogue, project-env-config, local-inheritance, stable-ciphertext, worktree-key): 204 pass, 0 fail. The run before it was at load average ~68 on the shared host and had 3 five-second timeouts in tests unrelated to this change (for example a whitespace-only key file is refused). The rerun passed.
  • bun run --cwd packages/cli typecheck: clean. bun run --cwd packages/cli check: 0 errors (34 existing warnings). Biome on every changed file: clean.
  • gitleaks 8.30.1 with the repo config and CI's --log-opts='--full-history --diff-filter=tuxdb HEAD': 915 commits, no leaks.
  • No CLI surface or guidance change in the rework, so the CLI reference and plugin renders are unchanged.

Negative controls

Every new test was proven with a mutation. Mutants killed:

  • unreadable_layer guard removed
  • reader-key priority swapped (environmentKey ?? fileKey)
  • tracking_unknown check removed
  • checkout marker always reported as present (8 tests fail)
  • secret_downgrade guard removed
  • JSON duplicate check removed

JSON scanner mutants, also all killed:

  • root does not expect a key
  • a comma at any depth expects a key
  • any opening brace expects a key
  • a key is not consumed
  • escapes are ignored
  • keys are compared without decoding
  • the scan does not skip to the string's end

Two mutants survived and were found to be equivalent: a redundant depth check and a nested-open reset. Both were removed from the code. The original change's 17 mutants were not rerun; the code they target did not change.

A secret-scan negative control used the base branch's config with no fingerprint: the PR range still reports 1 leak, from 06f2623. The current test file scans clean on its own.

Tests use synthetic values, private HOME via privateCliEnv, and temp git repos. The tracking_unknown fixture corrupts the temp repo's own .git/index. No test touches Docker, the keychain or real config.

For the reviewer

  • Acceptance interpretation: the issue says "overrides survive". Local entries outside the input survive, but an imported key replaces an operator's deliberate --local override of that same key. This is disclosed in the HACK-P08 missingContract and in the docs, and the pattern card stays partial. Please confirm whether this meets the bar.
  • secret_downgrade is a new refusal with no override flag. The way out is --secret, or hack env unset <key> --local and then a plain import. hack env add without --secret still downgrades, as before.
  • tracking_unknown refuses on a machine without git when the project has a .git entry. This is deliberate: fail closed.

Remaining gaps

  • Hydration cannot tell an operator's deliberate local override of an imported key from a stale hydrated value. An imported key always replaces its local entry.
  • hack env set --local does not take the hydration lock. Only the pre-rename recheck guards against it, which narrows the window but does not close it.
  • A SIGKILLed hydration leaves a stale lock (reported with its path) and a staging file. Both are ignored by git, and the lock needs manual removal. SIGINT/SIGTERM of the real CLI while it holds the lock is not tested; only SIGKILL is.
  • --env omitted writes the base local layer, as hack env add does today, not the configured default overlay.
  • Host-bound gaps from the issue are not covered here: native credential approval, runtime-injection receipts on a real stack, and log inspection on a live runtime. No production hydration was run. Hosted CI on fba3e87c is pending.

🤖 Generated with Claude Code

roodboi and others added 7 commits October 19, 2026 11:50
setProjectEnvValue now keeps an entry's existing ciphertext when it already
decrypts to the requested value under the current key, so repeating
`hack env set --secret --value-stdin` with the same value leaves the tracked
env file byte-identical and git status clean. A different value, a plain
entry becoming secret, or an entry that does not open under the current key
is re-encrypted with a fresh IV.

Env layer writes now commit by rename from an exclusive, fsynced sibling, so
a write interrupted before the commit leaves the previous layer intact and
resolvable. Existing permission bits and symlink targets are preserved.

HACK-P08 moves from deferred to partial: per-key no-op detection and atomic
layer writes exist; a multi-key hydration import remains missing.

Refs HACK-1152

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…vior

A `--local` env write killed before its commit rename left a
`.hack.env*.<uuid>.tmp` staging file that git did not ignore, so a later
`git add -A` could commit machine-local values, and a tracked-layer orphan
left the tree dirty. The managed `.hack/.gitignore` now ignores
`.hack.env*.tmp`, and `hack doctor` flags a tracked staging file.

Tests now cover symlink preservation, staging cleanup after a failed
rename, and clean git status after a killed tracked or local write. The
argv-env-value recipe skips flags, so the P08 card can name
`hack env set <key> --local --value-stdin --secret`. Docs describe the
first-repeat normalization of hand-edited files and the rename edge cases.

Refs HACK-1152

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs HACK-1152

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add `hack env hydrate`, which reads KEY=value entries (or one JSON object
with --format json) from piped stdin or --file and writes them to the
machine-local env layer in one rename-committed write.

- Only the effective local layer is written; a git-tracked one is refused.
- An entry that already holds its value is kept (secret ciphertext and
  scalar YAML type), and a no-op import does not rewrite the file. A
  changed value gets new ciphertext with a fresh IV.
- Local entries outside the input, other scopes and removal markers for
  other keys are kept.
- A secret in the layer that does not open under the reader key, a
  key-file mint that would shadow HACK_ENV_SECRET_KEY, a held or stale
  hydration lock, and a layer changed by another writer (compare-and-swap
  recheck before the rename) all refuse without changing the layer.
- Input is bounded (1 MiB), strict and all-or-nothing; refusals are
  closed and value-free and name at most a line number.
- Unexpected positional arguments are now reported by count, not echoed,
  so `hack env hydrate KEY=value` cannot print the value.

Refs HACK-1158.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With stable ciphertext, setting an unchanged secret keeps the stored
ciphertext, so the shared cipher fixture returned identical values and the
ciphertext-only change tests no longer produced a change. Encrypt each
fixture value in its own store so every call is a fresh encryption.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gitleaks flagged the literal HACK_ENV_SECRET_KEY fixture as a generic API
key. Build it at runtime like the repo's other key-shaped fixtures and
record the fingerprint of the branch commit that still holds the literal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Refuse hydration as tracking_unknown when the project is inside a git
  checkout but git cannot report whether it tracks the local layer (exit
  128 from a safe.directory refusal or damaged index, or git missing).
  Outside any checkout, hydration still proceeds.
- Refuse a plain import (no --secret) of a key the target scope stores as
  a secret (secret_downgrade), instead of silently replacing ciphertext
  with plaintext.
- Refuse repeated top-level keys in --format json input (duplicate_key),
  matching the env format, instead of keeping the last value.
- Pin the unreadable_layer refusal (invalid YAML, wrong shape, wrong
  version) and the key-file-before-HACK_ENV_SECRET_KEY reader priority
  with regression tests; share one runtime-built synthetic key helper.
- docs/env.md lists every hydration refusal.

Refs: HACK-1158

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@blacksmith-sh

This comment has been minimized.

roodboi and others added 6 commits October 19, 2026 14:25
The rename-committed env write treated any read failure as an absent file,
so `hack env set` on a layer it could not read (mode 000, or root-owned
0600 from an earlier `sudo hack`) built a layer holding only the new key and
renamed it over the old one, dropping every other value. Rename also skips
the write-permission check an in-place write got, so a read-only or foreign
layer was silently replaced with a new owner.

writeTextFileAtomicIfChanged now classifies the target before staging:
only a missing path or a dangling symlink is absent. An existing target
that cannot be read, is not a regular file, is not writable, or whose owner
and group the staging file cannot take is refused with a closed,
value-free AtomicWriteRefusedError, and the file is left as it was. A
rewrite keeps the file's owner and group as well as its mode.

Document the refusals and that rename detaches hard links, and list the
staging pathspec in the doctor check docstring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve src/lib/fs.ts: keep F7's target inspection, which refuses an
unreadable, non-regular, unwritable or unownable target instead of
reading it as absent, and apply F8's expectedExisting compare-and-swap
to the inspected text. The pre-rename check also uses the inspection, so
a target that became unreadable is refused rather than matching an
expected absence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve plugins/hack/skills/hack-cli/references/patterns.md by
regenerating it (bun run generate:agent-plugins) from the merged
src/agents/pattern-catalogue.ts, carrying this branch's pattern wording
and the changes merged in train b18.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The F7 branch now includes next, which switched one private-file write in
project-env-config.ts to publishPrivateFileExclusive and dropped the
createPrivateFileExclusive import. This branch's hydration lock still
creates its lock file exclusively, so import both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve generated plugin conflicts (plugins/hack/skills/hack-cli/references/patterns.md ) by regenerating
them with bun run generate:agent-plugins from the merged canonical source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve generated plugin conflicts (plugins/hack/skills/hack-cli/references/patterns.md ) by regenerating
them with bun run generate:agent-plugins from the merged canonical source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@blacksmith-sh

This comment has been minimized.

The owner_not_preserved refusal in writeTextFileAtomicIfChanged had no
test, because a non-root user cannot create a file owned by someone else.
A disposable child now reports the target as owned by another uid, so the
staging file's fchown really fails with EPERM. The test expects the
closed refusal, the old bytes and inode kept, and no staging file left.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Base automatically changed from wf/f7-stable-ciphertext-b16 to next October 21, 2026 22:42
roodboi and others added 2 commits October 21, 2026 18:45
…8-hydrate-b16

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#340 is now squash-merged into next (9a80dbd), so next and the F7 head
(e99f68c) hold the same F7 text. Conflicts were resolved against that
head as the effective base, keeping F8's changes and next's other work:

- docs/env.md, src/agents/pattern-catalogue.ts,
  src/lib/project-env-config.ts: F8's text. Next's side was F7's text,
  which F8 had already extended (hydrate pointer, P08 card, CAS
  expectedExisting option).
- src/cli/command.ts: keep #359's single-valued passthrough refusal
  exactly; other extra positionals are still reported by count only.
- plugins/.../patterns.md: next's version, then regenerated (output
  equals F8's render).
- src/lib/fs.ts merged without a conflict marker but duplicated
  writeTextFileAtomicIfChanged, AtomicWriteRefusedError and
  inspectAtomicWriteTarget; restored F8's single CAS-aware copy, which
  next's F7 version is a subset of.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@roodboi
roodboi marked this pull request as ready for review October 22, 2026 00:46
@roodboi
roodboi merged commit 7bebe9c into next Oct 22, 2026
16 checks passed
@roodboi
roodboi deleted the wf/f8-hydrate-b16 branch October 22, 2026 00:46
@linear-code

linear-code Bot commented Oct 22, 2026

Copy link
Copy Markdown

HACK-1158

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant