[roodboi]

Problem

A project with 20 services and one 64 KiB global secret fails hack up with
E_UNEXPECTED: E2BIG ... posix_spawn docker before Docker starts. Projects with 4 or 12
services work.

Root cause

#357 keeps decrypted Compose env values off disk: .hack/.internal/compose.env.override.yml
holds only ${HACK_ENV_*} references, and Hack passes the values to docker compose
through its process environment. renderComposeEnvReferenceOverride (in
src/lib/compose-managed-env.ts) gave every service its own reference for every key,
including keys that come from the global scope. The Compose child environment therefore
grew with services times keys: 20 × 64 KiB ≈ 1.25 MiB, plus the plain global copy, is over
the 1 MiB ARG_MAX on macOS. The spawn failure then reached the top-level handler as an
unclassified error.

Fix

• Shared global references. When a key's winning scope for a service is global, that
  service uses one composeEnvGlobalReferenceName({ key }) reference, shared with every
  other such service. A key that a service scope sets keeps its own per-service reference.
  The choice uses the resolved config's effectiveMetadata (names and scopes only), never
  the values, so the override text stays value-free. As a guard, a key that the metadata
  marks global but whose value differs from the global value keeps its own reference. The
  shared name keeps the HACK_ENV_<12 hex>_KEY shape, so withoutComposeEnvReferences
  still strips it from non-Compose children. Its digest input (\0global\0KEY) cannot
  equal a per-service input, including one for a service named global.
• Preflight with a closed error. The new src/lib/spawn-env-limit.ts measures the
  environment that the docker compose child would get: the live process.env plus the
  Compose env, with the NAME=value strings, their NULs and pointer slots. It checks that
  total against the platform limit and keeps 64 KiB for argv. The limits are 1 MiB on
  macOS. On Linux the limit is the kernel's formula, max(min(stack/4, 6 MiB), 128 KiB),
  with the stack soft limit read from /proc/self/limits (8 MiB is assumed when that file
  cannot be read), plus MAX_ARG_STRLEN = 128 KiB for one string. Other platforms get
  256 KiB. resolveModernComposeEnvOverrides and the legacy contract path run the check
  before they write the override or start Compose. This covers up, restart, run and
  exec. An oversized environment fails with the new code E_ENV_TOO_LARGE. Its message
  names the limit (ARG_MAX or MAX_ARG_STRLEN) and says how to reduce the environment.
  The --json detail is {limit, bytes, limitBytes}: byte counts only, with no names and
  no values.
• Adoption compatibility. LegacyComposeAdoptionProjection.resolve accepts three forms
  of a saved generation: the shared form, the earlier per-service reference form, and the
  literal form that only tests still model. A generation saved before this change still
  loads.
• Docs and guidance. docs/env.md replaces the "grows with services times keys"
  paragraph with the shared-reference behavior, the limits and the new code. The new code
  is registered in HACK_ERROR_CODES and in src/agents/error-code-reference.ts, and the
  plugin's hack-diagnose/references/error-codes.md was regenerated with
  bun run generate:agent-plugins. The CLI surface and package.json scripts did not
  change, so no CLI reference or task-catalog re-pin was needed.

Values stay off disk and out of argv and logs. The new code paths handle only byte
counts, names and scopes.

Tests

• tests/compose-env-spawn-size-cli.test.ts is the regression test. It runs the source
  CLI against a stub docker with a private HOME, an isolated Docker client config and a
  temp project, and makes no Docker socket, keychain or launchd calls.
  • 20 services, 64 KiB global secret: up --detach --json succeeds. The stub sees
    exactly 2 variables that carry the value (the shared reference and the plain global
    name), and the whole environment is under 256 KiB. All 20 services reference the same
    name, and the file, argv and output hold no value. The test assembles the values at
    runtime and checks them only with boolean includes, so a failing assertion never
    prints one.
  • Oversized environment: three 200 KiB global values give E_ENV_TOO_LARGE, exit 1,
    ARG_MAX on macOS or MAX_ARG_STRLEN on Linux. No docker compose runs and no
    override is written.
• tests/compose-env-shared-references.test.ts covers the renderer: sharing, a
  service-scope override (including one restated with the global value), the fallback
  service, value-independence, the metadata/value guard, the form without metadata, and
  name separation.
• tests/spawn-env-limit.test.ts covers measurement, the per-platform limits, the
  /proc/self/limits parser, the boundaries, the closed error's content and the inherited
  environment.
• tests/native-compose-adoption-projection.test.ts now expects the shared form from the
  current writer.

Mutation proof. Each mutation was applied alone, and the named tests failed:

Mutation	Caught by
sharing disabled	CLI 20-service test + 2 renderer tests
preflight removed from project.ts	CLI E_ENV_TOO_LARGE test
metadata not passed in project.ts	CLI 20-service test
scope check ignored	renderer "service-scoped keys"
value guard removed	renderer "disagrees"
pointer bytes dropped	measure test
MAX_ARG_STRLEN check removed	2 limit tests
per-service form no longer accepted in adoption	v3 value-free env override binds… (generation test)
shared form not accepted in adoption	real canonical legacy writers… (projection test)
inherited env ignored	2 limit tests

Pre-fix reproduction on macOS: with sharing disabled and the preflight removed, the
20-service CLI test exits 1 (the E2BIG path).

Evidence

• Bun 1.4.2 (mise x bun@1.4.2), pinned.
• Local (this Mac, which was under load): bun run --cwd packages/cli typecheck and root
  bun run check (including privacy:check) are clean, with only existing complexity
  warnings. Focused files pass: 168 tests across 13 files, plus
  native-compose-adoption-generation.test.ts at 114/114 when run alone. During one
  combined run that also included the generation file, that file had several tests fail
  at about 20 s each under host load. They pass alone.
• gitleaks 8.30.1 over origin/next..HEAD reports no leaks.
• Full suite on the M3 at 40c5581c: not completed. The run got the M3 CI lock at 11:34, but the M3 had only 56 GiB free, which is below the 58 GiB floor. I stopped it during the suite and removed its worktree. The M3 disk guard now holds the lock until space recovers. The hosted test job, which runs the full suite, passed at this exact head (see the rework section). An M3 full-suite run at 40c5581c is still an open gate.

Rework after the first CI run

Hosted CI at 78be3539 failed test and docker-e2e shards 3 and 4 with the same
closed error, "Adoption managed fixture source check failed; values omitted."

• Cause. The managed adoption fixture
  (tests/e2e/scenarios/native-compose-adoption-managed-inputs.ts) rebuilt the
  expected Compose env override without scope metadata. It therefore expected one
  reference per service and refused the shared global references that the writer now
  produces. The product code was right; the fixture oracle was stale. My local focused
  set had not included the fixture tests.
• Fix (f8f51a6c). The oracle now reads the same resolved scope metadata as the
  writer (resolveProjectEnvConfig(...).effectiveMetadata) and passes it as
  effectiveScopes.
• Mutation proof. With effectiveScopes removed from the writer in project.ts, the
  managed fixture uses real writer owners… test fails; restored, it passes.
• Merge of next (40c5581c). fix(adoption): report caller-deadline expiry apart from binding drift and refuse publishing a literal-bound override #397 changed the same adoption projection block.
  The resolution keeps the shared and per-service reference forms and takes fix(adoption): report caller-deadline expiry apart from binding drift and refuse publishing a literal-bound override #397's
  HACK-1318 comment. fix(adoption): report caller-deadline expiry apart from binding drift and refuse publishing a literal-bound override #397's literal-override classifier accepts the shared names,
  because they keep the ${NAME} reference shape.
• Evidence after the rework.
  • Hosted CI at 40c5581c: all 16 checks pass, including test and docker-e2e
    shards 1–4 (the native-compose-adoption-managed-worktrees and
    native-compose-adoption-local-worktrees scenarios).
  • Local: CLI typecheck and root bun run check (privacy-check: ok) are clean.
    Focused files pass, 73 tests across 10 files, including both fixture tests.
    generate:agent-plugins shows no drift. gitleaks 8.30.1 over
    origin/next..HEAD reports no leaks.
  • native-compose-adoption-generation.test.ts run alone on this Mac at load
    average 113: its first test timed out at 60 s and the rest of the file
    cascaded. The hosted test job passed it at this head.

Gaps

• On Linux the preflight trusts /proc/self/limits. A container runtime with a different
  stack limit for docker than for hack, or a kernel with non-4 KiB pages
  (MAX_ARG_STRLEN scales with page size), could still give a different real limit. The
  check is conservative on MAX_ARG_STRLEN (it assumes 4 KiB pages).
• The 64 KiB argv reserve is an estimate. A command line with thousands of service names
  could still exceed it in theory.
• Values still appear twice in the Compose environment (the shared reference and the
  plain global name used for ${KEY} interpolation). This is bounded, but it is not one
  copy.
• The CLI regression test has no real-Docker run. Hosted CI runs it on macOS and Linux.
• No M3 full-suite run at 40c5581c (see Evidence).

Refs HACK-1317

🤖 Generated with Claude Code

[linear-code]

HACK-1317