Skip to content

fix(runtime): give detached Compose image build/pull its own budget - #354

Merged
roodboi merged 1 commit into
nextfrom
wf/g6-compose-build-budget-b17
Oct 20, 2026
Merged

roodboi merged 1 commit into
nextfrom
wf/g6-compose-build-budget-b17

Conversation

@roodboi

@roodboi roodboi commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Refs HACK-1274

Root cause

Detached hack up / hack restart ran a single docker compose up -d inside one owned process group, bounded by HACK_COMPOSE_STARTUP_TIMEOUT_MS (default 90 s). That one budget covered image pull, image build, container creation, container start, and depends_on waiting. On a first-time Omega start (hack-cli up --path omega --detach, no local images), Compose was still pulling qdrant/qdrant and building omega-file-processor / omega-temporal-file-processor when the 90 s deadline killed the process group:

ERROR docker compose up exceeded its detached Compose startup budget of 90000 ms (HACK_COMPOSE_STARTUP_TIMEOUT_MS). ...

With HACK_COMPOSE_STARTUP_TIMEOUT_MS=2400000, the same start built the images and started 10 services. The budget meant for start and readiness was being used up by image acquisition.

Fix

Detached Compose startup now runs as two bounded invocations. Each one runs in its own owned process group and gets exit 124 when its deadline passes:

Phase Invocation Budget variable Default Range
images: image build/pull and container creation up --no-start [--force-recreate] [--no-deps] [services] HACK_COMPOSE_IMAGE_TIMEOUT_MS (new) 30 min 1 s – 4 h
start: start and dependency wait up -d --no-build [--no-deps] [services] HACK_COMPOSE_STARTUP_TIMEOUT_MS (unchanged) 90 s 1 s – 1 h
  • Recreation happens once, in the image phase. --no-build stops the start phase from rebuilding.
  • RuntimeUpOptions gains imageTimeoutMs and an onStartupPhase callback. up, full restart, the restart repair attempt, and targeted restart resolve both budgets before any hook or teardown, so an invalid value is refused early, as before. They also record which phase was running when the command failed.
  • E_STARTUP_TIMEOUT names the phase, its effective budget, and the variable to raise, for example ... startup budget of 90000 ms (phase: start and dependency wait). Raise it with HACK_COMPOSE_STARTUP_TIMEOUT_MS=<ms> ... or ... image build/pull budget of 1800000 ms (phase: image build/pull and container creation). Raise it with HACK_COMPOSE_IMAGE_TIMEOUT_MS=<ms> .... With --json, error.detail carries composePhase, budgetMs, and budgetEnv. A non-timeout failure stays E_COMPOSE_FAILED and its message names the phase.
  • The dependency-cache initializer timeout message now names its phase and HACK_COMPOSE_STARTUP_TIMEOUT_MS. Its budget is unchanged (see Gaps).
  • Foreground up is still a single unbounded invocation. down keeps its fixed 90 s limit.
  • docs/cli.md documents both phases, both variables, the JSON detail fields, and the pull_policy: always caveat.

Tests

  • tests/compose-startup-phases.test.ts (new) uses the real run() process-group deadline with a stub docker on PATH. The stub models Compose image handling: it builds missing images, --no-build refuses when images are missing, and -d can hang on a grandchild. No engine, socket, or real HOME is used.
    • Slow build with fast readiness succeeds: a 1.6 s build under a 1 s start budget exits 0, with phases images then start.
    • A real readiness hang still times out in the named phase: exit 124 in under 5 s with a 1 s start budget and a 10 s image budget. The hanging grandchild is gone, which shows the process-group kill. The message names start and dependency wait and HACK_COMPOSE_STARTUP_TIMEOUT_MS.
    • A hung build times out in the image phase without starting: exit 124, start is never invoked, the grandchild is killed, and the message names HACK_COMPOSE_IMAGE_TIMEOUT_MS.
  • tests/project-up-startup-state.test.ts covers up, restart, and scoped restart. It checks that the image budget is forwarded, that an image-phase or start-phase timeout is attributed to the right phase with the right variable, that an invalid image budget is refused before hooks or backend effects, and the JSON composePhase / budgetMs / budgetEnv fields.
  • tests/compose-startup-budget.test.ts covers the image budget's default, range, and independence from the startup budget. tests/runtime-backend.test.ts checks the exact two-invocation argv and the per-phase timeouts.
  • Mutation checks, each reverted afterwards:
    1. Back to a single up -d under the startup budget: the slow-build test fails with 124 at about 1 s, which reproduces the Omega failure.
    2. Start phase given the image budget: the readiness-hang test fails (10 s instead of under 5 s).
    3. Phase reports ignored in project.ts: four image-phase attribution and JSON tests fail.
    4. Image budget not resolved in project.ts: nine forwarding, attribution, and early-refusal tests fail.
  • A first stub flake was caused by the first exec of a newly written script taking more than 1 s on macOS. The stub is now run once before the timed calls, and 30 consecutive runs passed.
  • Checks run: the focused suites plus related project-*, lifecycle-json, and cli-result tests (222 pass across 21 files), tests/cli-spawn-env-isolation.test.ts, bun run --cwd packages/cli typecheck (0 errors), bun run --cwd packages/cli check (0 errors, only existing warnings), privacy:check, and docs:cli-reference (no diff; the CLI surface is unchanged).
  • Manual real-engine smoke on OrbStack with a throwaway Compose project, calling the backend directly and then removing everything it created:
    • Base next with a 5 s start budget and a build taking about 8 s: code: 124 at 5010 ms, during Building.
    • This branch with the same setup: code: 0. The image phase took about 12 s, then the app container reached Healthy.
    • This branch with a health check that never passes and a 3 s start budget: code: 124 in the start phase at about 3.7 s, with no Compose process left behind.

Gaps

  • I did not rerun this on the M3 against Omega. Proof comes from the stub tests and the local OrbStack smoke only.
  • Each detached start now makes one extra Compose invocation. In the smoke, a warm image phase with nothing to build took about 0.7 s, including creating two containers. I did not benchmark this precisely.
  • Services with pull_policy: always still pull in the start phase, because Compose applies that policy on every up. Those pulls count against the start budget. This is documented.
  • Dependency-cache installers (run --rm --no-deps during bootstrap) still acquire their image inside the HACK_COMPOSE_STARTUP_TIMEOUT_MS budget. Only the message changed.
  • The native compose and native authored runtime paths still use HACK_COMPOSE_STARTUP_TIMEOUT_MS as one aggregate deadline. They are out of scope here.
  • tests/native-compose-adoption-generation.test.ts fails on this host for both base next and this branch. Its first static7 test reaches the 60 s test timeout, and the later tests in the file fail after it. This failure is not related to this change.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Detached startup ran one `docker compose up -d` under the 90 s
HACK_COMPOSE_STARTUP_TIMEOUT_MS budget, so a first-time image build or
large pull was killed before any container started. Split detached
startup into an image phase (`up --no-start`, HACK_COMPOSE_IMAGE_TIMEOUT_MS,
default 30 min) and a start phase (`up -d --no-build`, the existing
startup budget). Each phase keeps its own owned process-group deadline, and
E_STARTUP_TIMEOUT now names the phase, its budget and the variable that
raises it.

Refs HACK-1274

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Oct 19, 2026

Copy link
Copy Markdown

HACK-1274

@roodboi
roodboi marked this pull request as ready for review October 20, 2026 13:58
@roodboi
roodboi merged commit b431f0e into next Oct 20, 2026
16 checks passed
@roodboi
roodboi deleted the wf/g6-compose-build-budget-b17 branch October 20, 2026 13:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant