Repository navigation
fix(runtime): give detached Compose image build/pull its own budget - #354
Merged
Merged
Conversation
Detached startup ran one `docker compose up -d` under the 90 s HACK_COMPOSE_STARTUP_TIMEOUT_MS budget, so a first-time image build or large pull was killed before any container started. Split detached startup into an image phase (`up --no-start`, HACK_COMPOSE_IMAGE_TIMEOUT_MS, default 30 min) and a start phase (`up -d --no-build`, the existing startup budget). Each phase keeps its own owned process-group deadline, and E_STARTUP_TIMEOUT now names the phase, its budget and the variable that raises it. Refs HACK-1274 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs HACK-1274
Root cause
Detached
hack up/hack restartran a singledocker compose up -dinside one owned process group, bounded byHACK_COMPOSE_STARTUP_TIMEOUT_MS(default 90 s). That one budget covered image pull, image build, container creation, container start, anddepends_onwaiting. On a first-time Omega start (hack-cli up --path omega --detach, no local images), Compose was still pullingqdrant/qdrantand buildingomega-file-processor/omega-temporal-file-processorwhen the 90 s deadline killed the process group:With
HACK_COMPOSE_STARTUP_TIMEOUT_MS=2400000, the same start built the images and started 10 services. The budget meant for start and readiness was being used up by image acquisition.Fix
Detached Compose startup now runs as two bounded invocations. Each one runs in its own owned process group and gets exit 124 when its deadline passes:
images: image build/pull and container creationup --no-start [--force-recreate] [--no-deps] [services]HACK_COMPOSE_IMAGE_TIMEOUT_MS(new)start: start and dependency waitup -d --no-build [--no-deps] [services]HACK_COMPOSE_STARTUP_TIMEOUT_MS(unchanged)--no-buildstops the start phase from rebuilding.RuntimeUpOptionsgainsimageTimeoutMsand anonStartupPhasecallback.up, fullrestart, the restart repair attempt, and targetedrestartresolve both budgets before any hook or teardown, so an invalid value is refused early, as before. They also record which phase was running when the command failed.E_STARTUP_TIMEOUTnames the phase, its effective budget, and the variable to raise, for example... startup budget of 90000 ms (phase: start and dependency wait). Raise it with HACK_COMPOSE_STARTUP_TIMEOUT_MS=<ms> ...or... image build/pull budget of 1800000 ms (phase: image build/pull and container creation). Raise it with HACK_COMPOSE_IMAGE_TIMEOUT_MS=<ms> .... With--json,error.detailcarriescomposePhase,budgetMs, andbudgetEnv. A non-timeout failure staysE_COMPOSE_FAILEDand its message names the phase.HACK_COMPOSE_STARTUP_TIMEOUT_MS. Its budget is unchanged (see Gaps).upis still a single unbounded invocation.downkeeps its fixed 90 s limit.docs/cli.mddocuments both phases, both variables, the JSON detail fields, and thepull_policy: alwayscaveat.Tests
tests/compose-startup-phases.test.ts(new) uses the realrun()process-group deadline with a stubdockeronPATH. The stub models Compose image handling: it builds missing images,--no-buildrefuses when images are missing, and-dcan hang on a grandchild. No engine, socket, or real HOME is used.imagesthenstart.start and dependency waitandHACK_COMPOSE_STARTUP_TIMEOUT_MS.startis never invoked, the grandchild is killed, and the message namesHACK_COMPOSE_IMAGE_TIMEOUT_MS.tests/project-up-startup-state.test.tscoversup,restart, and scopedrestart. It checks that the image budget is forwarded, that an image-phase or start-phase timeout is attributed to the right phase with the right variable, that an invalid image budget is refused before hooks or backend effects, and the JSONcomposePhase/budgetMs/budgetEnvfields.tests/compose-startup-budget.test.tscovers the image budget's default, range, and independence from the startup budget.tests/runtime-backend.test.tschecks the exact two-invocation argv and the per-phase timeouts.up -dunder the startup budget: the slow-build test fails with 124 at about 1 s, which reproduces the Omega failure.project.ts: four image-phase attribution and JSON tests fail.project.ts: nine forwarding, attribution, and early-refusal tests fail.project-*,lifecycle-json, andcli-resulttests (222 pass across 21 files),tests/cli-spawn-env-isolation.test.ts,bun run --cwd packages/cli typecheck(0 errors),bun run --cwd packages/cli check(0 errors, only existing warnings),privacy:check, anddocs:cli-reference(no diff; the CLI surface is unchanged).nextwith a 5 s start budget and a build taking about 8 s:code: 124at 5010 ms, duringBuilding.code: 0. The image phase took about 12 s, then the app container reachedHealthy.code: 124in thestartphase at about 3.7 s, with no Compose process left behind.Gaps
pull_policy: alwaysstill pull in the start phase, because Compose applies that policy on everyup. Those pulls count against the start budget. This is documented.run --rm --no-depsduring bootstrap) still acquire their image inside theHACK_COMPOSE_STARTUP_TIMEOUT_MSbudget. Only the message changed.HACK_COMPOSE_STARTUP_TIMEOUT_MSas one aggregate deadline. They are out of scope here.tests/native-compose-adoption-generation.test.tsfails on this host for both basenextand this branch. Its first static7 test reaches the 60 s test timeout, and the later tests in the file fail after it. This failure is not related to this change.🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need.