From fe61654b76c6cc47f70351ad04f2c4819d9111b8 Mon Sep 17 00:00:00 2001 From: leecyang <164771957+leecyang@users.noreply.github.com> Date: Tue, 20 Oct 2026 12:06:01 +0000 Subject: [PATCH 1/3] build: trim unused musl Canvas binding from glibc server image --- .../issue-229-canvas-trim/FAILURE-MODEL.md | 11 ++++ docs/evidence/issue-229-canvas-trim/README.md | 48 ++++++++++++++ scripts/prune-glibc-canvas-runtime.mjs | 65 +++++++++++++++++++ server/docker/lingxiloop-server.Dockerfile | 2 + .../lingxiloop-server.Dockerfile.dockerignore | 1 + server/scripts/check-packaged-startup.mjs | 19 ++++++ 6 files changed, 146 insertions(+) create mode 100644 docs/evidence/issue-229-canvas-trim/FAILURE-MODEL.md create mode 100644 docs/evidence/issue-229-canvas-trim/README.md create mode 100644 scripts/prune-glibc-canvas-runtime.mjs diff --git a/docs/evidence/issue-229-canvas-trim/FAILURE-MODEL.md b/docs/evidence/issue-229-canvas-trim/FAILURE-MODEL.md new file mode 100644 index 000000000..c0a9bd46a --- /dev/null +++ b/docs/evidence/issue-229-canvas-trim/FAILURE-MODEL.md @@ -0,0 +1,11 @@ +# #229 glibc Canvas trim: failure model before implementation + +Scope is the **server production dependency stage** of the frozen `d1075e35` candidate, on `linux/amd64` Node 22 Bookworm/glibc. The SPA build stage and its dependencies stay complete. This is a separate draft; it does not change the published image or production. + +1. **Wrong platform or changed package.** A future musl base, non-x64 image, changed `@napi-rs/canvas` version, loader or native binaries must stop the build before deletion. Check `process.platform`, `process.arch`, glibc runtime, package versions, directory layout and the audited file SHA-256 values. +2. **Active GNU binary is accidentally removed or cannot load.** Delete only `@napi-rs/canvas-linux-x64-musl/skia.linux-x64-musl.node`; retain both package manifests, README files, the parent `@napi-rs/canvas` LICENSE, and the GNU binary. In separate Node processes before/after deletion, load Canvas, inspect which `.node` file was selected, and draw an actual PNG. +3. **PDF.js needs native Canvas through a dynamic path.** Run the packaged LingxiOS document-worker on the unchanged PDF fixture with the musl binary absent, compare exact output hash with the unmodified image, and retain the PDF.js parser, worker, CMaps, standard fonts and WASM. +4. **A future consumer requires musl portability or another libc.** The trim is fixed to the current glibc-only final runtime stage and guarded by the audited loader hash. Moving to musl or expanding image platforms requires a new audit. A build that cannot prove this condition fails. +5. **The apparent 30 MB gain is mistaken for whole-image savings.** Record regular bytes and reproducible single-file gzip only as an upper-level proxy; actual candidate OCI layer and full-image deltas require a completed build. Hardlinks and tar compression cannot be inferred from raw file size. + +Stop this round after either this single bounded candidate has a passing isolated short check and draft PR, or its checks fail. Other large packages remain documented candidates rather than speculative deletions. diff --git a/docs/evidence/issue-229-canvas-trim/README.md b/docs/evidence/issue-229-canvas-trim/README.md new file mode 100644 index 000000000..ed9f1b3fc --- /dev/null +++ b/docs/evidence/issue-229-canvas-trim/README.md @@ -0,0 +1,48 @@ +# #229 bounded remaining-slimming review (2026-10-20 UTC) + +This is a separate draft based on the coordinator's frozen product source `d1075e35e16721979b484c2d9e33eeb56468ce9d`, not a deployment. Its immutable `linux/amd64` image index is `sha256:697d5a3e3d34629e603c120cd068abde3a2add611f583d5444d77a947887bf69`. The [final OCI receipt](../release-215-216-219/image-final/verified-oci/receipt.json), [verified merged rootfs inventory](../release-215-216-219/image-final/verified-oci/rootfs-inventory.json) and [path analysis](../release-215-216-219/image-final/analysis/pr236-d107-rootfs-analysis.json) are inputs. The inventory file's locally rechecked SHA-256 is `3d537b85a17d2dd22790264678a83ce64e147f2f0192f4e2bfda40d666f8c96a`. No image layer was downloaded for this follow-up; the previously pulled historical image was used only for focused native/PDF checks. The final image is **published but not deployed** in the coordinator's current receipt. + +## Largest ten production dependency packages and consumers + +These are the ten largest **non-overlapping top-level npm package directories**, ranked by regular-file bytes in the final image. Together they contain 176,256,490 of the production dependency tree's 243,634,898 regular bytes (72.34%). Figures are uncompressed content, not OCI transfer bytes. Node itself is a separate 124,827,920-byte binary, and the SPA `/app/dist` is 66,290,700 regular bytes; neither is double-counted in this package ranking. + +| Rank | Package | Regular bytes | Actual consumer / deletion decision | +| ---: | --- | ---: | --- | +| 1 | `@napi-rs/canvas-linux-x64-gnu` | 33,975,707 | PDF.js legacy Node loader calls `@napi-rs/canvas`; on Bookworm/glibc it loads the GNU binding. The native 33,974,784-byte file is active; retain. | +| 2 | `@napi-rs/canvas-linux-x64-musl` | 30,316,535 | Same loader selects this branch only on musl. Final target is Bookworm/glibc x64; one 30,315,608-byte native file is the bounded candidate below. Keep package metadata; trim only the binary after a platform, version, loader and hash check. | +| 3 | `pdfjs-dist` | 24,272,468 | Formal LingxiOS 3.5.8 document-worker imports `legacy/build/pdf.mjs` and resolves CMaps/fonts/WASM. The prior audited viewer trim is already present; parser/worker/resources remain. Some active legacy maps need a separate diagnostics policy, so this pass retains them. | +| 4 | `js-tiktoken` | 22,432,828 | Openlit's `dist/helpers.js` requires it and calls `encodingForModel` with a fallback. Its CJS/ESM files and model data appear duplicated, but package conditional exports and other model families have not been qualified for per-file deletion. | +| 5 | `openai` | 12,468,421 | `server/src/llm-client.ts` constructs the shared provider SDK client. Removing it would remove current model calls. | +| 6 | `@opentelemetry/semantic-conventions` | 12,006,437 | Openlit's `dist/manual-trace.js` imports it; product initializes Openlit when an OTLP endpoint is configured. Retain telemetry semantics and current sensitive-content settings. | +| 7 | `@opentelemetry/exporter-metrics-otlp-http` | 11,971,470 | Openlit's metrics module constructs the OTLP metrics exporter; removing it changes configured observability. | +| 8 | `@esbuild/linux-x64` | 11,428,465 | `npm` starts Web/Worker/migrate through `tsx`, whose runtime transform imports `esbuild` and selects this binary. `esbuild/bin/esbuild` is a hardlink to this file, not another 11.4 MB copy. | +| 9 | `@opentelemetry/otlp-transformer` | 9,638,682 | The OTLP exporter uses its serializer. The lock contains some nested OpenTelemetry versions; replacing them or deleting alternate formats requires API/version and live telemetry qualification. | +| 10 | `@dicebear/styles` | 7,745,477 | Server `profile-avatar.ts` imports `marbles.json` and `planets.json` for generated user/course avatars. Other styles may be another small package-specific candidate, but not a proved safe whole-package removal. | + +The final compressed OCI layer budget is 96,721,079 bytes of base/runtime-APT layers, 70,347,423 bytes of production dependencies, and 37,786,863 bytes of SPA dist. These three disjoint areas already total **204,855,365 of 205,959,232 bytes (99.46%)**. Node, Python, bubblewrap/tini, GNU Canvas, the TypeScript launcher, PDF parsing, provider SDK, telemetry when configured, and the full SPA have present consumers. Source-only trimming cannot produce a large whole-image percentage. The remaining 6,050 dependency source maps (43,257,857 raw bytes), test/fixture-path candidates, old avatars and alternate package formats are *not* collectively proven removable; the categories overlap and cannot be summed as savings. + +## One proved platform-only trim and bounded verification + +The final OCI inventory reports the exact same `@napi-rs/canvas@1.0.8` loader and GNU/musl binary SHA-256 values as the previously pulled, immutable `ac906946` image. A no-network scan of 18,009 code/JSON/TS files in that exact historical image found five references to the musl package: two lock files, two package manifests and the Canvas loader, with no product code caller. Frozen d107 product source also has no direct musl-package import. The loader's Linux x64 branch checks libc and loads **GNU** on glibc; the final Dockerfile's dependency and runtime stages use Node 22 Bookworm slim. The new script deletes only `node_modules/@napi-rs/canvas-linux-x64-musl/skia.linux-x64-musl.node` in the **server production dependency stage**. It aborts before deletion on another platform, libc, version, layout or hash. The package manifests, README files, parent Canvas `LICENSE`, GNU binary, PDF.js and SPA stage are untouched. The final-image packaged check also requires a working GNU Canvas render and confirms the musl binary is absent. [Failure model](FAILURE-MODEL.md) was written before the source change. + +Measured candidate file: **30,315,608 regular bytes**, SHA-256 `00790d510c990e1972f33d3c587d3d9faccddd4bf49c2bbe1938a05c40844034`; fixed single-file `gzip -n -9` is **12,496,273 bytes**. This gzip number is a rough compressed-size proxy, **not** the new dependency layer or whole-image delta. A full candidate image has not been built. The unchanged old image's Canvas loader hash, GNU/musl binary hashes and versions match the final OCI inventory exactly; it is a native-library surrogate, not an exact final-image business test. + +Checks already run without network or production access: + +- Exact historical image Node 22 dry run selected GNU and reported the one target file. A task-owned package copy with a deliberately changed musl version caused `--apply` to exit 1 while the binary remained. Restoring the version and running `--apply` against that copy removed the one binary; package metadata and GNU remained. +- With the pruned package copy read-only-mounted into a separate container, Canvas drew a PNG and `require.cache` contained only the GNU `.node`. The real LingxiOS document-worker processed `e2e/homework/triangle-reference.pdf` before and after: identical SHA-256 `30ee46cdb0878604ece1e00fab7a9807f3db64ff443a48e6dc2fe4a5646a5e04` (201 output bytes, as in the prior PDF audit). +- `node --check` for both changed scripts and `git diff --check` passed. A local two-file source test was **8 pass, 1 fail** because this checkout's installed LingxiOS is stale **3.5.2** while frozen source/lock requires **3.5.8**; the failed import is `consumePublicTextAttempts`, unrelated to Canvas. The frozen d107 upstream CI passed before this draft, but it does not certify this change. Exact-branch CI with a clean dependency install is required. + +## Size limit and stop condition + +If the isolated 12,496,273-byte gzip proxy translated one-for-one into an OCI-layer reduction, the whole image would be about **193,462,959 bytes**, or **6.47% below** the historical 206,838,733-byte image. This is an illustration, not a candidate OCI measurement or guarantee. The retained base/APT plus SPA layers alone are **134,507,942 compressed bytes**. A hypothetical 32% whole-image reduction from the historical image would target 140,650,338 bytes, leaving only 6,142,396 bytes for **all** runtime dependencies if those two areas stayed fixed; the current dependency layer is 70,347,423 bytes and contains several proven runtime consumers. Thus the prior ~32% *source COPY* reduction cannot be extended into a safe 32% whole-image claim through small dependency deletions. + +This round stops with the one pinned, functional platform-only candidate. Further savings from source maps, telemetry versions, tokenizer formats, avatars or SPA assets need their own consumer and regression audits; a base/runtime or SPA redesign is a separate product change. No source, license, isolation support, diagnostic map or other runtime resource is removed to chase a percentage. #229 still requires controlled build/disk measurements and final configured acceptance; this draft does not close it. + +## Feasible cold/warm build and peak-disk measurement, not run here + +Use a dedicated ephemeral **linux/amd64** nonproduction builder with `overlay2`, at least **80 GiB free before starting**, a task-owned scratch filesystem, and no unrelated Docker workloads. The current executor has about 20 GB free and Docker `vfs`; no full candidate build is attempted here. Before either run, record `df -B1`, Docker root path/driver, Buildx version, frozen source SHA, image platform, build args, base-image digest, npm lock hashes, and the APT/npm repository or snapshot identities. Hold all those inputs fixed; if the APT feed cannot be frozen, label the result accordingly. + +Create a dedicated empty BuildKit builder. Build the frozen source once into a task-owned OCI archive (cold), then repeat the identical command with the **same builder/cache** into a second archive (warm), using the existing npm secret mount without printing its value. Capture monotonic wall time, `--progress plain` cache-hit lines, archive digest/size, and exit status for each. Sample `df -B1` on the Docker-root/scratch filesystem every second from before build start through export completion; peak disk is the largest increase in used bytes over the exclusive runner's baseline. Also record `docker system df --verbose` before/after for attribution, but do not substitute that endpoint value for the sampled peak. Abort only the task-owned build if free space falls below 20 GiB; retain logs and mark the pair incomplete. Repeat cold/warm pairs on clean ephemeral runners to report spread rather than treating a single CI `build/push` step duration as a controlled comparison. Existing 328-second publication step and 1.68 GB root dev-install observation do not measure this pair or its disk peak. + +No production cleanup, service stop, data/backup deletion, image rollback, merge, deploy or issue closure was performed. diff --git a/scripts/prune-glibc-canvas-runtime.mjs b/scripts/prune-glibc-canvas-runtime.mjs new file mode 100644 index 000000000..7d3c154e6 --- /dev/null +++ b/scripts/prune-glibc-canvas-runtime.mjs @@ -0,0 +1,65 @@ +// Keep the Debian/glibc Canvas binding used by server-side PDF.js. The pinned +// optional musl binding cannot be selected by this exact platform/loader. +// Default is a dry run; the Docker deps stage supplies --apply. +import assert from 'node:assert/strict' +import { createHash } from 'node:crypto' +import { existsSync, lstatSync, readFileSync, readdirSync, unlinkSync } from 'node:fs' +import { join, resolve } from 'node:path' +import { createRequire } from 'node:module' + +const [rootArg, action] = process.argv.slice(2) +assert.ok(rootArg && (action === undefined || action === '--apply'), + 'Usage: node prune-glibc-canvas-runtime.mjs NODE_MODULES [--apply]') +const root = resolve(rootArg) +const packagePath = name => join(root, '@napi-rs', name) +const parent = packagePath('canvas') +const gnu = packagePath('canvas-linux-x64-gnu') +const musl = packagePath('canvas-linux-x64-musl') +const gnuBinary = join(gnu, 'skia.linux-x64-gnu.node') +const muslBinary = join(musl, 'skia.linux-x64-musl.node') +const loader = join(parent, 'js-binding.js') +const sha256 = path => createHash('sha256').update(readFileSync(path)).digest('hex') +const version = path => JSON.parse(readFileSync(join(path, 'package.json'), 'utf8')).version + +assert.equal(process.platform, 'linux', 'Canvas trim requires Linux') +assert.equal(process.arch, 'x64', 'Canvas trim requires x64') +assert.ok(process.report.getReport().header.glibcVersionRuntime, + 'Canvas trim requires the current glibc runtime') +for (const path of [parent, gnu, musl]) { + assert.ok(lstatSync(path).isDirectory(), `Expected installed package directory: ${path}`) + assert.equal(version(path), '1.0.8', `Canvas package changed: ${path}`) +} +assert.deepEqual(readdirSync(musl).sort(), + ['README.md', 'package.json', 'skia.linux-x64-musl.node'].sort(), + 'musl package layout changed; review before trimming') +assert.ok(existsSync(join(parent, 'LICENSE')), 'Missing Canvas license') +assert.equal(sha256(loader), 'be59c8d4888630cf88656f65b63dd150e60bdb1d5eab0ba5b47c0a6cb389533e', + 'Canvas libc loader changed; review branch selection') +assert.equal(sha256(gnuBinary), '738cf404fc3422bd1f57dae002f19b2f4f41d2dd5614e8023e0b64d7ef3a8e10', + 'GNU binding changed; review native runtime') +assert.ok(lstatSync(muslBinary).isFile(), 'Expected musl binding file') +assert.equal(sha256(muslBinary), '00790d510c990e1972f33d3c587d3d9faccddd4bf49c2bbe1938a05c40844034', + 'musl binding changed; review before trimming') + +// Run the installed loader on the actual build platform before deletion. +const require = createRequire(join(root, '..', 'package.json')) +const { createCanvas } = require('@napi-rs/canvas') +const canvas = createCanvas(1, 1) +canvas.getContext('2d').fillRect(0, 0, 1, 1) +assert.ok(canvas.toBuffer('image/png').length > 0, 'Canvas drawing failed') +assert.ok(Object.hasOwn(require.cache, gnuBinary), 'Canvas did not load the GNU binding') +assert.ok(!Object.hasOwn(require.cache, muslBinary), 'Canvas unexpectedly loaded the musl binding') + +const receipt = { + platform: 'linux/x64/glibc', packageVersion: '1.0.8', + removedPath: 'node_modules/@napi-rs/canvas-linux-x64-musl/skia.linux-x64-musl.node', + regularBytes: lstatSync(muslBinary).size, + sha256: '00790d510c990e1972f33d3c587d3d9faccddd4bf49c2bbe1938a05c40844034', + applied: action === '--apply', +} +if (action === '--apply') { + unlinkSync(muslBinary) + assert.ok(!existsSync(muslBinary), 'musl binding remains after trim') + assert.ok(existsSync(gnuBinary), 'GNU binding missing after trim') +} +console.log(JSON.stringify(receipt)) diff --git a/server/docker/lingxiloop-server.Dockerfile b/server/docker/lingxiloop-server.Dockerfile index 73d98c4d9..35bc5e286 100644 --- a/server/docker/lingxiloop-server.Dockerfile +++ b/server/docker/lingxiloop-server.Dockerfile @@ -32,10 +32,12 @@ WORKDIR /app COPY server/package.json server/package-lock.json server/.npmrc ./ COPY package.json /tmp/qualified-package.json COPY scripts/prune-pdfjs-server-runtime.mjs /tmp/prune-pdfjs-server-runtime.mjs +COPY scripts/prune-glibc-canvas-runtime.mjs /tmp/prune-glibc-canvas-runtime.mjs RUN node --input-type=module -e 'import fs from "node:fs"; import assert from "node:assert/strict"; const expected=JSON.parse(fs.readFileSync("/tmp/qualified-package.json")).dependencies["@lyyzka/lingxios"]; assert.match(expected,/^\d+\.\d+\.\d+$/); assert.equal(JSON.parse(fs.readFileSync("package.json")).dependencies["@lyyzka/lingxios"],expected,"server runtime must match root qualification");' RUN --mount=type=secret,id=npm_token \ NODE_AUTH_TOKEN="$(cat /run/secrets/npm_token 2>/dev/null)" npm ci --registry="${NPM_REGISTRY}" --omit=dev --no-audit --no-fund --prefer-offline \ && node /tmp/prune-pdfjs-server-runtime.mjs /app/node_modules/pdfjs-dist --apply \ + && node /tmp/prune-glibc-canvas-runtime.mjs /app/node_modules --apply \ && node --input-type=module -e 'for (const name of ["pdfjs-dist", "pdfjs-dist/legacy/build/pdf.mjs"]) { const { getDocument } = await import(name); if (typeof getDocument !== "function") throw Error(`Broken PDF.js entry: ${name}`); }' RUN node --input-type=module -e 'import fs from "node:fs"; import assert from "node:assert/strict"; import {releaseVersions} from "@lyyzka/lingxios"; assert.equal(releaseVersions.runtime,JSON.parse(fs.readFileSync("/tmp/qualified-package.json")).dependencies["@lyyzka/lingxios"],"installed image runtime must match root qualification");' diff --git a/server/docker/lingxiloop-server.Dockerfile.dockerignore b/server/docker/lingxiloop-server.Dockerfile.dockerignore index 24fb1f9f9..8a994b34f 100644 --- a/server/docker/lingxiloop-server.Dockerfile.dockerignore +++ b/server/docker/lingxiloop-server.Dockerfile.dockerignore @@ -24,6 +24,7 @@ !scripts/verify-zcode-visualize.mjs !scripts/build-visualize-runtime.mjs !scripts/prune-pdfjs-server-runtime.mjs +!scripts/prune-glibc-canvas-runtime.mjs # Don't bake source maps / built assets / huge logs into the layer. server/**/*.log server/**/dist diff --git a/server/scripts/check-packaged-startup.mjs b/server/scripts/check-packaged-startup.mjs index 640d64155..0990ee60f 100644 --- a/server/scripts/check-packaged-startup.mjs +++ b/server/scripts/check-packaged-startup.mjs @@ -3,6 +3,25 @@ import { spawnSync } from 'node:child_process' import { existsSync, readFileSync, readdirSync } from 'node:fs' import { join } from 'node:path' import { fileURLToPath } from 'node:url' +import { createRequire } from 'node:module' + +// The final stage is Debian/glibc x64. Confirm the native GNU Canvas path +// still works after the optional musl-only binary has been removed. +const canvasRoot = 'node_modules/@napi-rs' +assert.ok(process.report.getReport().header.glibcVersionRuntime, 'Expected glibc runtime') +assert.equal(process.arch, 'x64', 'Expected linux/amd64 Canvas runtime') +assert.equal(existsSync(join(canvasRoot, 'canvas-linux-x64-musl/skia.linux-x64-musl.node')), false, + 'Unexpected musl Canvas binary in glibc image') +assert.equal(existsSync(join(canvasRoot, 'canvas-linux-x64-gnu/skia.linux-x64-gnu.node')), true, + 'Missing GNU Canvas binary') +assert.equal(existsSync(join(canvasRoot, 'canvas/LICENSE')), true, 'Missing Canvas license') +const require = createRequire(import.meta.url) +const { createCanvas } = require('@napi-rs/canvas') +const canvas = createCanvas(1, 1) +canvas.getContext('2d').fillRect(0, 0, 1, 1) +assert.ok(canvas.toBuffer('image/png').length > 0, 'Canvas drawing failed') +assert.ok(Object.hasOwn(require.cache, join(process.cwd(), canvasRoot, + 'canvas-linux-x64-gnu/skia.linux-x64-gnu.node')), 'Canvas did not load GNU native binding') // The server deps stage removes only PDF viewer assets and unused generic maps. // Keep future product imports from silently depending on those removed paths. From d1341cba3facda5976185cddac6d582f29cf8638 Mon Sep 17 00:00:00 2001 From: leecyang <164771957+leecyang@users.noreply.github.com> Date: Tue, 20 Oct 2026 14:53:10 +0000 Subject: [PATCH 2/3] feat: add guarded same-schema host releases [skip deploy] --- .github/workflows/ci.yml | 8 +- docs/host-release-failure-model.md | 36 +++++ docs/host-release.md | 127 +++++++++++++++ scripts/host-release-compose.mjs | 29 ++++ scripts/host-release-komodo.mjs | 149 ++++++++++++++++++ scripts/host-release-komodo.test.mjs | 92 +++++++++++ scripts/host-release-transport.test.mjs | 35 +++++ scripts/host-release.local.test.mjs | 57 +++++++ scripts/host-release.mjs | 201 ++++++++++++++++++++++++ scripts/host-release.test.mjs | 88 +++++++++++ scripts/schema-upgrade-komodo.mjs | 40 ++++- 11 files changed, 858 insertions(+), 4 deletions(-) create mode 100644 docs/host-release-failure-model.md create mode 100644 docs/host-release.md create mode 100644 scripts/host-release-compose.mjs create mode 100644 scripts/host-release-komodo.mjs create mode 100644 scripts/host-release-komodo.test.mjs create mode 100644 scripts/host-release-transport.test.mjs create mode 100644 scripts/host-release.local.test.mjs create mode 100644 scripts/host-release.mjs create mode 100644 scripts/host-release.test.mjs diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 7343fee30..89a9ec4cc 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -196,6 +196,7 @@ jobs: - 'scripts/trigger-komodo-rollout.mjs' - 'scripts/prepare-forum-rollout.mjs' - 'scripts/schema-upgrade-*.mjs' + - 'scripts/host-release*.mjs' - 'scripts/verify-memory-backup*.mjs' - 'scripts/sync-komodo-config.mjs' - 'scripts/test-distributed-realtime.mjs' @@ -204,6 +205,7 @@ jobs: - 'scripts/trigger-komodo-rollout.mjs' - 'scripts/prepare-forum-rollout.mjs' - 'scripts/schema-upgrade-*.mjs' + - 'scripts/host-release*.mjs' - 'scripts/verify-memory-backup*.mjs' - 'scripts/sync-komodo-config.mjs' - 'deploy/**' @@ -315,7 +317,9 @@ jobs: working-directory: third_party/open-notebook run: uv run --frozen --python 3.12 pytest -q - if: matrix.component == 'deploy-contract' - run: node --test scripts/production-rag-contract.test.mjs scripts/release-smoke.test.mjs scripts/schema-upgrade-maintenance.test.mjs scripts/schema-upgrade-host-source.test.mjs + run: node --test scripts/production-rag-contract.test.mjs scripts/release-smoke.test.mjs scripts/schema-upgrade-maintenance.test.mjs scripts/schema-upgrade-host-source.test.mjs scripts/host-release.test.mjs scripts/host-release-transport.test.mjs scripts/host-release-komodo.test.mjs + - if: matrix.component == 'deploy-contract' + run: node --test --test-name-pattern 'effective candidate' scripts/host-release.local.test.mjs - if: matrix.component == 'release' run: npm run version:check @@ -532,6 +536,6 @@ jobs: KOMODO_API_SECRET: ${{ secrets.KOMODO_API_SECRET }} steps: - uses: actions/checkout@v5 - with: { ref: main } + with: { ref: '${{ needs.update-manifests.outputs.commit-sha }}' } - name: Trigger Komodo rollout run: node scripts/trigger-komodo-rollout.mjs diff --git a/docs/host-release-failure-model.md b/docs/host-release-failure-model.md new file mode 100644 index 000000000..8049f8af9 --- /dev/null +++ b/docs/host-release-failure-model.md @@ -0,0 +1,36 @@ +# Same-schema host release: failure model (written before implementation) + +This is an application-only release transaction, separate from native12→13 maintenance. It requires an accepted source, exactly the same formal runtime package/integrity, native schema, protocol, schema hash and all 48 product migration checksums. It never runs migrations, backup/restore, Git, a broad procedure, ResourceSync or infrastructure deployment. + +## Failures and required boundaries + +1. A caller relabels a different schema/package/inventory as compatible: compare both committed manifests/locks, formal installed package, image metadata on both hosts and the actual read-only database state. Reject before source writes or stops. +2. Wrong source acceptance, image audit or plan bytes: bind their SHA256 values; require passed receipts, exact source/image identities and successful exact-target CI qualification. +3. Source files or config drift, stale Core cache, wrong server, running procedure/migration, automatic deployment or unknown hooks: require available exact hosts and Stack IDs, native cache refresh without errors, exact per-stack file hashes and complete config readback. Core's source and policy remain unchanged. +4. Target Compose changes environment, security, volumes, commands or infrastructure: compare effective, non-interpolated Compose against the accepted source, permitting only immutable app image pins and `pull_policy`. Require `never` for all three target app services. +5. Docker image-store ID differs from OCI config digest: use actual per-host InspectImage.Id, bind audited index/RepoDigest, linux/amd64 and ordered RootFS diffIDs, then require each container's Image to equal its host's verified ID. Never guess the ID from config or index. +6. Image missing/drifted: normal release fails before staging or stopping. No implicit pull occurs during release. An explicit online-only preparation mode may stage a separately committed pull template, PullStack exactly `db-migrate`, validate its terminal receipt and actual image, then restore accepted source bytes. It never stops/deploys services; every live container identity/health remains unchanged. Partial/unknown preparation stops without retry or implicit recovery. +7. Compose attempts registry access or starts db-migrate: app target policy is precisely `auto_pull=false`, `extra_args=["--no-deps","--pull","never"]`, all three effective services `pull_policy: never`. Komodo RunStackService's `pull:false` only disables its explicit pull; the template policy is necessary for its `compose run`. Core's unchanged SQL oneoff retains its existing image/pull policy and remains an explicit availability dependency. +8. Partial host write or config update, lost response, wrong update operation/target: record submitting/known update ID before progressing, require terminal success and exact readback. Never retry unknown outcomes, never overwrite the prior receipt, never deploy after a failed stage. +9. Worker stop accepted but still running, identity replaced, live native leases/attempts/capture/outbox or prepared transactions: fail before any new Web. Both old Workers stop first; require two fresh zero-lease observations. Web connections and queued work are allowed; this is not a database-wide no-client barrier. Establish image qualification and re-inspect the original containers before the first source write. Once a target service becomes healthy, retain its exact container identity; a later replacement with the same tag is not silently accepted. +10. Web can remain available while Workers drain. Update Web A, Web B, Worker A, Worker B, verifying exact image and actual health each time. Recheck unchanged schema throughout. After new Worker A begins, its legitimate leases must not be confused with old-worker leakage; all old Workers must remain stopped/replaced. +11. Target unhealthy or policy changes: stop the sequence. Unknown operation => no further network action; known failure => only a fresh read-only compatibility check for recovery classification, never automatic rollback. +12. Recovery: preserve public source bytes/identities, original container evidence, stage and operation receipts. Only a separately reviewed recovery may restore prior public templates and redeploy the accepted compatible image after resolving every known update and proving the exact DB inventory again. No old-schema restore, no credential or environment export, no automatic replay of a partial receipt. +13. Two executors: exclusive local receipt creation and recorded ownership are necessary, but not a distributed lock. Native action/procedure idle gates are checked before every mutation. The designated sole production executor must serialize manual/CI entrypoints; this driver does not claim cross-host locking. +14. Receipt/log leakage: no config/environment, raw host source, API reply/log, credential, prompt or memory body enters progress/errors. Native WriteStackFileContents logs its argument, so send only exact tracked public Compose templates. + +## Public command contracts + +Komodo v2.3.3 [`bin/periphery/src/api/compose.rs`](https://github.com/moghtech/komodo/blob/v2.3.3/bin/periphery/src/api/compose.rs) (SHA256 `883bb3387a2a320f02831f61b6325817faa1f935f6cff3ecbf1f4c43161ea8ee`) issues a separate `compose pull` only when `auto_pull` is true, and builds `compose ... up -d -- `. It does not itself add `--no-deps`. Core exposes PullStack; PullImage is an internal Periphery API and is not used. + +Docker documents `up --no-deps` and `up --pull never`; `run` independently supports a pull policy. Real local Compose tests must prove that a cached image with `pull_policy: always` starts via `up --no-deps --pull never` without touching an unavailable registry or starting its dependency, and that a missing image fails without pulling. The metadata oneoff must use a template with effective `pull_policy: never`. + +Sources: https://docs.docker.com/reference/cli/docker/compose/up/ and https://docs.docker.com/reference/cli/docker/compose/run/ ; Komodo tag https://github.com/moghtech/komodo/tree/v2.3.3 . + +App B has an existing gateway service. The effective-Compose comparison retains it byte-for-byte at the parsed configuration level; neither the release nor image-preparation service selection includes it. + +Local Compose2.40.3 uses `--no-TTY`/`-T`, whereas the reviewed Komodo2.3.3 builds `--no-tty`. Production's existing successful oneoff receipts establish compatibility there. The local oneoff test records its equivalent `-T` usage and does not claim native long-flag compatibility on2.40.3. The `up` flags are exercised exactly. + +## Delivery boundary + +The ordinary trigger remains fail-closed for host/pinned stacks. This first driver is explicit and does not call the legacy rollout. Persistent ResourceSync declarations and ordinary CI dispatch need a separately sealed integration before any generic rollout is reenabled. No production execution is part of local development/tests. diff --git a/docs/host-release.md b/docs/host-release.md new file mode 100644 index 000000000..b70b20611 --- /dev/null +++ b/docs/host-release.md @@ -0,0 +1,127 @@ +# Explicit same-schema host release + +Use `scripts/host-release.mjs` only after the preceding production release has a sealed successful acceptance. This application-only path keeps Komodo's native host-file source, leaves core-state and app B's gateway unchanged, and does not call the legacy rollout procedure. The normal trigger still rejects unexplained host/pinned state. + +The initial implementation deliberately supports native13/product48 only. Source and target must have identical runtime version, published package tarball/integrity, protocol, native schema/hash and all 48 product migration names/checksums. It does not import the application migrator. Web A, Web B, Worker A, Worker B are the only deploy targets. + +## Qualification and plan + +All input paths below are absolute. A plan is sealed from a descriptor by the offline default mode. The CLI independently checks the accepted source receipt, image audit bytes, Git objects, four package manifests/locks, migration files, effective Compose and locally installed formal package. Use the already qualified dependency installation; no installation, image build, credential discovery or remote request is performed by plan mode. + +Descriptor fields: + +| Field | Contract | +| --- | --- | +| `format` | Exactly `same-schema-host-release/v1`; this is not a schema-upgrade plan. | +| `createdAt` | ISO timestamp. | +| `source`, `target` | `releaseDescriptor(commit, formalRelease)` output plus `image` below. | +| `sourceAcceptanceSha256` | SHA256 of the previous successful acceptance JSON bytes. It must have `passed:true`, exact source `candidate`, and every named check true. | +| `qualification` | `{commit: target.sha, conclusion:"success", ciUrl:"https://github.com/LingXi-Org/LingxiLoop/actions/runs/"}` from verified exact-target CI. | +| `stacks` | Ordered app A, app B, core-state records below. | +| `planHash` | Created by offline sealing; never hand-edit a sealed plan. | + +Each `image` is `{indexDigest,configDigest,manifestDigest,diffIds,auditSha256}`. Populate from the actual passed full linux/amd64 OCI audit: `manifestDigest=linuxAmd64ManifestDigest`, `diffIds=layers.map(layer=>layer.diffId)`, and `auditSha256` hashes its exact JSON bytes. Image audit files must match the candidate source and digest identities. Docker's actual per-host `InspectImage.Id` is established online; it is not inferred from the OCI config digest. + +Each stack record is: + +```json +{ + "stack": "lingxiloop-app-a", + "server": "shanghai-a", + "source": { + "commit": "", + "branch": "release/", + "sha256": "", + "autoPull": true, + "extraArgs": [] + }, + "target": { + "commit": "", + "branch": "release/", + "sha256": "", + "autoPull": false, + "extraArgs": ["--no-deps", "--pull", "never"] + }, + "preparation": { + "commit": "", + "sha256": "" + } +} +``` + +App B uses `shanghai-b`. Core uses `shanghai-a`, has no `preparation`, and its entire source and target records must be equal. Record actual accepted source policy; do not assume all previous releases used `autoPull:true`. + +The target app templates must pin all three app services (`db-migrate`, `lingxiloop`, `worker`) to the exact target image and use effective `pull_policy: never`. The preparation templates have the same target pin and `pull_policy: always`. They may differ from the accepted templates only in those image/pull-policy fields. Environment, security, volumes, commands, ports, resource limits and gateway configuration must be unchanged. Raw templates must be regular tracked Git files at their exact per-stack manifest commits. Target manifest commits require `[skip deploy]`. + +Use separate commits for accepted source, temporary pull policy and final never policy as needed. These are public source artifacts; do not render environment substitutions or copy production `.env` into them. Native `WriteStackFileContents` records its contents in Komodo logs. + +```sh +node scripts/host-release.mjs \ + --input /absolute/descriptor.json --output /absolute/plan.json \ + --acceptance /absolute/source-acceptance.json \ + --source-audit /absolute/source-image-audit.json \ + --target-audit /absolute/target-image-audit.json +``` + +The plan records exact-head CI qualification; the designated release owner must obtain its successful GitHub receipt. An arbitrary URL in a descriptor is not evidence of CI success. Production execution remains gated by the owning release review, exact source acceptance and image audit. + +## Image cache preparation + +Normal `--execute` requires both target images already cached and qualified before staging or stopping. Missing/drifted images fail closed. + +Komodo2.3.3 has no public Core server PullImage action. The driver does not call its private Periphery counterpart. If explicit online preparation is needed: + +```sh +node scripts/host-release.mjs --prepare-images \ + --input /absolute/plan.json --output /absolute/new-image-preparation-receipt.json \ + --acceptance /absolute/source-acceptance.json \ + --source-audit /absolute/source-image-audit.json \ + --target-audit /absolute/target-image-audit.json +``` + +This uses only existing injected `KOMODO_API_KEY`/`KOMODO_API_SECRET`. It keeps all four accepted containers healthy and unchanged, stages the qualified pull template on one app stack, invokes native `PullStack` for `db-migrate` only, verifies exact terminal receipt and cached index/platform/layers, then restores the original accepted template. It repeats for the other host. It does not update stack config, deploy services or migrate. It always pulls when explicitly selected; do not select it if normal cached-image qualification already passes. + +A partial or uncertain preparation does not restore/retry automatically. Preserve its receipt, resolve the known update, inspect the exact remaining host files, then review the narrow restoration of accepted public bytes. Normal release refuses a leftover preparation template. + +## Release and acceptance + +```sh +node scripts/host-release.mjs --execute \ + --input /absolute/plan.json --output /absolute/new-release-receipt.json \ + --acceptance /absolute/source-acceptance.json \ + --source-audit /absolute/source-image-audit.json \ + --target-audit /absolute/target-image-audit.json +``` + +The receipt is exclusively created and records every known update ID. Only one designated executor may own a production release. Native action/procedure guards and exclusive local files are not a distributed release lock. + +The driver first verifies actual source DB and images, stages exact target public files, makes the narrow app-only provenance/`auto_pull`/`extra_args` update with full config readback, and runs actual DB-free target metadata on both hosts. Both old Workers then stop with bounded grace. Two read-only observations must show zero live work/session/attempt/capture/outbox leases and no prepared transaction. Web connections and queued work are allowed. + +It deploys Web A, Web B, Worker A, Worker B with exact host source, image and health checks. It checks compatible DB provenance throughout and retains each newly healthy target's container identity. It does not require new Worker A's legitimate leases to be zero before starting Worker B. + +`auto_pull=false` suppresses Komodo's separate pre-deploy pull. `extra_args=["--no-deps","--pull","never"]` suppresses Compose dependency creation and pulling during `up`; all three services' `pull_policy: never` also covers metadata `run`, whose public Komodo request does not expose a pull-policy override. Core's unchanged read-only SQL oneoff retains its current source/image policy and is still an availability dependency. + +A `complete` driver receipt establishes the ordered transaction's actual source/image/state/health checks. The release owner must still seal external HTTPS version, readiness, dependency health and audited public-asset comparisons, and any available authenticated smoke, before declaring the new release accepted. + +## Recovery and permanent integration + +There is no automatic rollback/resume command. Unknown submission, failed source readback or acknowledgement without exact config proof stops all further requests. Known failures may perform one fresh read-only state check to classify whether same-schema recovery remains reviewable; that classification never starts a service. + +For reviewed recovery, first resolve every saved update and confirm the failed executor is dead. Verify exact current DB provenance and all partial host/config/container identities. Drain any new Workers before restoring the previous compatible image. Restore only reviewed public source bytes and narrow source-policy fields through the native APIs, then deploy in the same service order with cached-image/health checks. Seal a new receipt that retains the original failed one. No database restore, native downgrade, credential/security/environment change or broad procedure is authorized by this driver. + +Do not reenable the legacy ordinary pipeline after using this path. Its existing preflight rejects host mode before sync/deployment. A later integration must declare per-stack host mode in desired resources, allowlist ResourceSync changes, route ordinary application releases through this driver and serialize release ownership. This change fixes CI's manifest checkout to the exact `update-manifests` output, but intentionally does not run or unlock the old broad procedure. + +## Owning checks + +```sh +node --test scripts/host-release.test.mjs scripts/host-release-transport.test.mjs scripts/host-release-komodo.test.mjs +node --test --test-name-pattern 'effective candidate' scripts/host-release.local.test.mjs +``` + +The real local Docker contract additionally requires an already cached `redis:7-alpine` image and a Docker daemon: + +```sh +node --test scripts/host-release.local.test.mjs +``` + +It only retags that existing image under an unreachable loopback registry and uses a unique local Compose project, cleaning those exact resources afterward. It performs no image build or registry pull. It writes `/tmp/host-release-compose-contract.json`; local Compose2.40.3's `-T` spelling limitation is explicitly recorded. See [the failure model](host-release-failure-model.md). diff --git a/scripts/host-release-compose.mjs b/scripts/host-release-compose.mjs new file mode 100644 index 000000000..7da00a37f --- /dev/null +++ b/scripts/host-release-compose.mjs @@ -0,0 +1,29 @@ +import {spawnSync} from 'node:child_process' +import {isDeepStrictEqual as equal} from 'node:util' +const IMAGE='accel.way2api.fun/ghcr.io/lingxi-org/lingxiloop-server:' +const services=['db-migrate','lingxiloop','worker'] +export function parsePublicCompose(source){ + if(typeof source!=='string'||Buffer.byteLength(source)>512*1024)throw Error('Invalid public Compose source') + const r=spawnSync('docker',['compose','--env-file','/dev/null','--project-name','lingxiloop-host-preflight','--file','-','config','--no-interpolate','--no-env-resolution','--no-path-resolution','--format','json'], + {input:source,encoding:'utf8',env:{PATH:process.env.PATH},timeout:10000,killSignal:'SIGKILL',maxBuffer:1024*1024}) + if(r.error||r.status!==0)throw Error('Public Compose validation failed') + try{return JSON.parse(r.stdout)}catch{throw Error('Public Compose validation failed')} +} +export function assertHostComposeChange(source,target,sourceSha,targetSha,pullPolicy='never'){ + const before=parsePublicCompose(source),after=parsePublicCompose(target) + const names=Object.keys(before.services??{}).sort() + if(!equal(names,Object.keys(after.services??{}).sort())||!services.every(s=>names.includes(s))||names.some(s=>![...services,'gateway'].includes(s)))throw Error('Unexpected application services') + for(const service of services){ + const a=before.services[service],b=after.services[service] + if(a.image!==IMAGE+sourceSha||b.image!==IMAGE+targetSha||b.pull_policy!==pullPolicy||a.build||b.build)throw Error('Image pin/pull policy differs') + // Compare everything except the two intentionally changed image fields. + a.image=b.image;a.pull_policy=b.pull_policy + } + // Compose retains extension anchors; compare their non-image policy too. + for(const config of [before,after])for(const [k,v] of Object.entries(config))if(k.startsWith('x-')&&v&&typeof v==='object'){ + if('image'in v)v.image=IMAGE+targetSha + if('pull_policy'in v)v.pull_policy=pullPolicy + } + if(!equal(before,after))throw Error('Release changes environment/security/volumes or unrelated Compose policy') + return true +} diff --git a/scripts/host-release-komodo.mjs b/scripts/host-release-komodo.mjs new file mode 100644 index 000000000..547e5a6d7 --- /dev/null +++ b/scripts/host-release-komodo.mjs @@ -0,0 +1,149 @@ +// Native Komodo2.3.3 calls. No Git/private API, migration, secret or env writes. +import {isDeepStrictEqual as equal} from 'node:util' +import {setTimeout as delay} from 'node:timers/promises' +import {createTransport,parseMarker,validateStateReceipt,stateCommand,METADATA_CODE} from './schema-upgrade-komodo.mjs' +import {APPS,CORE,IMAGE,RELEASE_ARGS,check,hash,validateHostPlanShape,assertImageIdentity} from './host-release.mjs' +const actions=['pulling','deploying','starting','restarting','pausing','unpausing','stopping','destroying'] +const id=/^[a-f0-9]{24}$/,hex=/^[a-f0-9]{64}$/ +const oneOff=(stack,service,extra)=>({stack,service,no_tty:true,no_deps:true,detach:false,service_ports:false,pull:false,...extra}) +export function createHostReleaseOperations({plan,files,apiKey,apiSecret,signal,onProgress=async()=>{},transport,wait=delay}){ + validateHostPlanShape(plan) + const t=transport??createTransport({apiKey,apiSecret,signal,onProgress}) + const phases=new Map(plan.stacks.map(r=>[r.stack,'source'])),configs=new Map(),identities=new Map(),images=new Map(),containerPolicies=new Map() + const rowFor=stack=>{const r=plan.stacks.find(r=>r.stack===stack);check(r,'Unexpected stack');return r} + const imageKey=(which,server)=>which+':'+server + async function hosts(){ + const result=await t.read('ListServers',{});check(Array.isArray(result),'Server inventory unavailable') + for(const name of ['manager','shanghai-a','shanghai-b'])check(result.filter(r=>r.name===name&&r.info?.state==='Ok'&&id.test(r.id??'')).length===1,'Required host unavailable') + return result + } + function assertConfig(detail,row,serverId){ + const c=detail?.config,phase=phases.get(row.stack),ref=phase==='target'?row.target:row.source,emptyHook={path:'',command:'',shell_mode:false} + check(id.test(detail?._id?.$oid??'')&&c&&c.server_id===serverId&&c.swarm_id===''&&c.repo==='LingXi-Org/LingxiLoop' + &&c.branch===ref.branch&&c.commit===ref.commit&&c.files_on_host===true&&c.reclone===false&&c.clone_path==='' + &&c.git_account===''&&c.registry_account===''&&c.linked_repo===''&&c.file_contents==='' + &&c.run_directory===`deploy/komodo/${row.stack}`&&c.env_file_path==='.env'&&equal(c.file_paths,['compose.yml']) + &&equal(c.additional_env_files,[])&&equal(c.config_files,[])&&c.auto_update===false&&c.auto_update_all_services===false + &&c.poll_for_updates===false&&c.auto_pull===ref.autoPull&&equal(c.extra_args,ref.extraArgs)&&c.run_build===false + &&c.destroy_before_deploy===false&&c.project_name===''&&c.compose_cmd_wrapper===''&&equal(c.build_extra_args,[]) + &&equal(c.pre_deploy,emptyHook)&&equal(c.post_deploy,emptyHook)&&equal(c.ignore_services,row.stack===CORE?[]:['db-migrate']),'Frozen source/configuration drift') + if(identities.has(row.stack))check(identities.get(row.stack)===detail._id.$oid,'Stack identity changed') + else identities.set(row.stack,detail._id.$oid) + if(configs.has(row.stack))check(equal(c,configs.get(row.stack)),'Full stack configuration changed') + else configs.set(row.stack,structuredClone(c)) + } + async function guard(stacks=plan.stacks.map(r=>r.stack)){ + signal?.throwIfAborted() + check((await t.read('GetProcedureActionState',{procedure:'lingxiloop-production-rollout'})).running===false,'Concurrent rollout procedure') + for(const stack of stacks){ + const row=rowFor(stack),servers=await hosts(),server=servers.find(r=>r.name===row.server) + const action=await t.read('GetStackActionState',{stack});check(actions.every(k=>action[k]===false),'Concurrent stack action') + const before=await t.read('GetStack',{stack});assertConfig(before,row,server.id) + await t.refreshStackCache(stack) + const after=await t.read('GetStack',{stack}),freshServer=(await hosts()).find(r=>r.name===row.server) + check(freshServer.id===server.id,'Host identity changed');assertConfig(after,row,server.id) + check(equal(before.config,after.config),'Source configuration changed while refreshing') + const info=after.info,remote=info?.remote_contents + check(equal(info?.remote_errors,[])&&equal(info?.missing_files,[])&&Array.isArray(remote)&&remote.length===1&&remote[0].path==='compose.yml'&&typeof remote[0].contents==='string','Fresh host source unavailable') + const phase=phases.get(stack),ref=phase==='target-file'?row.target:phase==='preparation'?row.preparation:row[phase] + check(hash(remote[0].contents)===ref.sha256,'Host Compose hash differs from sealed source') + await onProgress({phase:'host-source',status:'verified',stack,sourcePhase:phase,sha256:ref.sha256}) + } + } + async function assertIdle(){ + await guard() + for(const stack of APPS){ + const services=await t.read('ListStackServices',{stack});check(Array.isArray(services),'Service inventory unavailable') + const jobs=services.filter(r=>r.service==='db-migrate');check(jobs.length<=1,'Ambiguous migration job') + if(jobs[0]?.container){ + check(hex.test(jobs[0].container.id??''),'Migration container identity unavailable') + const c=await t.read('InspectContainer',{server:rowFor(stack).server,container:jobs[0].container.id}) + check(c.Id===jobs[0].container.id&&c.State?.Running===false&&c.State?.Restarting===false,'Existing migration job unresolved') + } + } + } + async function inspect(){ + const result=[] + for(const stack of APPS){ + const row=rowFor(stack),services=await t.read('ListStackServices',{stack});check(Array.isArray(services),'Service inventory unavailable') + for(const service of ['lingxiloop','worker']){ + const matches=services.filter(s=>s.service===service);check(matches.length===1&&hex.test(matches[0].container?.id??''),'Ambiguous application service') + const c=await t.read('InspectContainer',{server:row.server,container:matches[0].container.id}) + const env=c.Config?.Env,source=env?.find(x=>x.startsWith('LINGXILOOP_COMMIT_SHA='))?.slice('LINGXILOOP_COMMIT_SHA='.length) + check([plan.source.sha,plan.target.sha].includes(source)&&c.Id===matches[0].container.id&&c.Config?.Image===IMAGE+source + &&/^sha256:[a-f0-9]{64}$/.test(c.Image??'')&&typeof c.State?.Running==='boolean'&&typeof c.State?.Restarting==='boolean' + &&Number.isSafeInteger(c.RestartCount)&&Number.isSafeInteger(c.State.Pid)&&c.State.Pid>=0&&c.State.OOMKilled===false,'Unexpected actual container identity/state') + const which=source===plan.source.sha?'source':'target',qualified=images.get(imageKey(which,row.server)) + if(qualified)check(c.Image===qualified,'Container image differs from actual qualified host image') + // Keep secret-bearing values only in memory. Never persist their contents. + const protectedConfig={env:env.filter(x=>!x.startsWith('LINGXILOOP_COMMIT_SHA=')).toSorted(),host:c.HostConfig,user:c.Config.User,command:c.Config.Cmd,entrypoint:c.Config.Entrypoint,health:c.Config.Healthcheck,mounts:c.Mounts} + // Docker fills container-specific host fields; compare security/volume fields + // explicitly instead of comparing the whole HostConfig object. + protectedConfig.host=Object.fromEntries(['ReadonlyRootfs','Privileged','SecurityOpt','CapAdd','CapDrop','PidsLimit','Binds','Tmpfs','PortBindings','Memory','NanoCpus','Devices','DeviceRequests'].map(k=>[k,c.HostConfig?.[k]])) + const key=stack+':'+service + if(containerPolicies.has(key))check(equal(containerPolicies.get(key),protectedConfig),'Actual environment/security/volumes/startup policy changed') + else containerPolicies.set(key,structuredClone(protectedConfig)) + result.push({stack,server:row.server,service,id:c.Id,source,imageId:c.Image,running:c.State.Running,restarting:c.State.Restarting,restartCount:c.RestartCount,pid:c.State.Pid,healthy:c.State.Health?.Status==='healthy', + policy:{readOnly:c.HostConfig?.ReadonlyRootfs,privileged:c.HostConfig?.Privileged,securityOpt:c.HostConfig?.SecurityOpt,capAdd:c.HostConfig?.CapAdd,capDrop:c.HostConfig?.CapDrop,pidsLimit:c.HostConfig?.PidsLimit,user:c.Config.User,productionMode:env.includes('NODE_ENV=production')}}) + } + } + return result + } + async function verifyImages(which='target',onlyStack){ + check(['source','target'].includes(which)&&(!onlyStack||APPS.includes(onlyStack)),'Unexpected image qualification scope') + const descriptor=plan[which] + for(const stack of onlyStack?[onlyStack]:APPS){ + const row=rowFor(stack),actual=await t.read('InspectImage',{server:row.server,image:IMAGE+descriptor.sha}),imageId=assertImageIdentity(actual,descriptor.image) + const key=imageKey(which,row.server) + if(images.has(key))check(images.get(key)===imageId,'Cached image changed during release') + images.set(key,imageId) + await onProgress({phase:'image',status:'verified',stack,release:which,source:descriptor.sha,imageId,indexDigest:descriptor.image.indexDigest,platform:'linux/amd64',layers:descriptor.image.diffIds.length}) + } + } + async function writeFile(stack,which,nextPhase){ + const row=rowFor(stack);check(APPS.includes(stack),'Core source writes prohibited') + await guard([stack]) + const contents=files?.get(stack)?.[which];check(typeof contents==='string'&&hash(contents)===row[which].sha256,'Unqualified public source bytes') + await t.writeHostFile(stack,contents);phases.set(stack,nextPhase) + try{await guard([stack])}catch{throw Object.assign(Error('Host write succeeded but exact source readback is unresolved'),{uncertain:true})} + } + async function stage(stack){ + check(phases.get(stack)==='source','Release staging may not replay a partial source') + await writeFile(stack,'target','target-file') + const row=rowFor(stack),before=structuredClone(configs.get(stack)),patch={commit:row.target.commit,branch:row.target.branch,auto_pull:false,extra_args:RELEASE_ARGS} + await t.setHostReleaseConfig(stack,patch) + try{ + const after=await t.read('GetStack',{stack}) + check(after?._id?.$oid===identities.get(stack)&&equal(after.config,{...before,...patch}),'Full post-update configuration mismatch') + configs.set(stack,structuredClone(after.config));phases.set(stack,'target');await guard([stack]) + }catch{throw Object.assign(Error('Configuration acknowledged but exact source/config readback is unresolved'),{uncertain:true})} + await onProgress({phase:'host-config',status:'verified',stack,fullConfigReadback:true}) + } + async function state(){ + await guard([CORE]) + return validateStateReceipt(parseMarker(await t.execute('RunStackService',oneOff(CORE,'postgres',{entrypoint:'sh',user:'postgres',command:['-ec',stateCommand]})),'STATE')) + } + async function metadata(stack){ + check(APPS.includes(stack)&&phases.get(stack)==='target','Metadata requires qualified target host source') + await guard([stack]) + return parseMarker(await t.execute('RunStackService',oneOff(stack,'db-migrate',{command:['timeout','-k','5','30','node','--input-type=module','-e',METADATA_CODE+`console.log('SCHEMA_UPGRADE_METADATA='+JSON.stringify(metadata));`]})),'METADATA') + } + return {assertIdle,inspect,verifyImages,stage,state,metadata, + async stop(stack,service){check(APPS.includes(stack)&&service==='worker'&&phases.get(stack)==='target','Only old Workers may be drained');await guard([stack]);return t.execute('StopStack',{stack,services:[service],stop_time:150})}, + async deploy(stack,service){check(APPS.includes(stack)&&['lingxiloop','worker'].includes(service)&&phases.get(stack)==='target','Only qualified app services may deploy');await guard([stack]);return t.execute('DeployStack',{stack,services:[service]})}, + async verifyHealthy(stack,service){ + const deadline=Date.now()+180000 + while(Date.now()r.stack===stack&&r.service===service) + check(r?.source===plan.target.sha&&r.imageId===images.get(imageKey('target',r.server)),'Deployed service is not qualified target') + if(r.running&&!r.restarting&&r.healthy&&r.restartCount===0)return + await wait(2000,undefined,{signal}) + } + throw Error('Application readiness deadline exceeded') + }, + async stagePreparation(stack){check(phases.get(stack)==='source','Image preparation cannot replay partial staging');await writeFile(stack,'preparation','preparation')}, + async pullPrepared(stack){check(APPS.includes(stack)&&phases.get(stack)==='preparation','Only explicit prepared image pull allowed');await guard([stack]);return t.execute('PullStack',{stack,services:['db-migrate']})}, + async restoreSource(stack){check(phases.get(stack)==='preparation','Expected online preparation source');await writeFile(stack,'source','source')}, + wait:ms=>wait(ms,undefined,{signal}), + } +} diff --git a/scripts/host-release-komodo.test.mjs b/scripts/host-release-komodo.test.mjs new file mode 100644 index 000000000..cf4241cd9 --- /dev/null +++ b/scripts/host-release-komodo.test.mjs @@ -0,0 +1,92 @@ +import test from 'node:test' +import assert from 'node:assert/strict' +import {createHostReleaseOperations} from './host-release-komodo.mjs' +import {APPS,CORE,IMAGE,RELEASE_ARGS,hash,sealHostPlan,runHostRelease,runImagePreparation} from './host-release.mjs' +function fixture(fault={}){ + const a='a'.repeat(40),b='b'.repeat(40),h='c'.repeat(64),d='sha256:'+h + const formal={runtime:'3.5.8',schema:13,protocol:11,schemaHash:h,integrity:'sha512-'+Buffer.alloc(64).toString('base64'),tarball:'https://npm.pkg.github.com/download/@lyyzka/lingxios/3.5.8/fixture',productVersion:48,migrations:Array.from({length:48},(_,i)=>({version:i+1,name:'migration_'+(i+1),checksum:h}))} + const image={indexDigest:d,configDigest:'sha256:'+'d'.repeat(64),manifestDigest:'sha256:'+'e'.repeat(64),diffIds:[d],auditSha256:h} + const source={sha:a,...formal,image},target={sha:b,...formal,image},files=new Map(),configs=new Map(),actualFiles=new Map(),events=[],calls=[] + const plan=sealHostPlan({format:'same-schema-host-release/v1',createdAt:'2026-10-20T00:00:00Z',source,target,sourceAcceptanceSha256:h,qualification:{commit:b,conclusion:'success',ciUrl:'https://github.com/LingXi-Org/LingxiLoop/actions/runs/1'},stacks:[...APPS,CORE].map((stack,i)=>{ + const source='public source '+stack,target=i===2?source:'public target '+stack,preparation='public pull '+stack + files.set(stack,{source,target,preparation});actualFiles.set(stack,source) + const s={commit:a,branch:'release/source',sha256:hash(source),autoPull:true,extraArgs:[]} + return {stack,server:i===1?'shanghai-b':'shanghai-a',source:s,target:i===2?s:{commit:b,branch:'release/target',sha256:hash(target),autoPull:false,extraArgs:RELEASE_ARGS},...(i===2?{}:{preparation:{commit:b,sha256:hash(preparation)}})} + })}) + for(const [i,row] of plan.stacks.entries())configs.set(row.stack,{_id:{$oid:String(i+1).repeat(24)},config:{ + server_id:row.server==='shanghai-a'?'a'.repeat(24):'b'.repeat(24),swarm_id:'',repo:'LingXi-Org/LingxiLoop',branch:row.source.branch,commit:a, + files_on_host:true,reclone:false,clone_path:'',git_account:'',registry_account:'',linked_repo:'',file_contents:'',run_directory:'deploy/komodo/'+row.stack, + env_file_path:'.env',file_paths:['compose.yml'],additional_env_files:[],config_files:[],auto_update:false,auto_update_all_services:false,poll_for_updates:false, + auto_pull:true,run_build:false,destroy_before_deploy:false,project_name:'',compose_cmd_wrapper:'',extra_args:[],build_extra_args:[],pre_deploy:{path:'',command:'',shell_mode:false},post_deploy:{path:'',command:'',shell_mode:false}, + ignore_services:row.stack===CORE?[]:['db-migrate'],environment:'private fixture never persist', + }}) + const containers=APPS.flatMap((stack,i)=>['lingxiloop','worker'].map((service,j)=>({ + stack,service,Id:String(i*2+j+1).repeat(64),Image:'sha256:'+(fault.sourceImageMismatch?'0':'f').repeat(64), + Config:{Image:IMAGE+a,Env:['LINGXILOOP_COMMIT_SHA='+a,'NODE_ENV=production','SECRET=private fixture never persist'],User:'node',Cmd:['npm','run',service==='worker'?'worker:start':'server:start']}, + State:{Running:true,Restarting:false,Pid:1,OOMKilled:false,Health:{Status:'healthy'}},RestartCount:0, + HostConfig:{ReadonlyRootfs:service==='worker',Privileged:false,SecurityOpt:[],CapAdd:[],CapDrop:[],PidsLimit:128}, + }))) + const state={...formal,installedSchema:13,product:48,count:48,readOnly:true,fullVisibility:true,liveWork:0,liveSession:0,liveAttempts:0,liveCapture:0,liveOutbox:0,prepared:0,otherClients:4,queued:3,pendingCapture:1} + delete state.integrity;delete state.tarball;delete state.productVersion + const t={ + async read(op,p){ + calls.push({op,...p}) + if(op==='GetProcedureActionState')return {running:false} + if(op==='ListServers')return ['manager','shanghai-a','shanghai-b'].map((name,i)=>({name,id:(i===2?'b':'a').repeat(24),info:{state:fault.hostDown&&name==='shanghai-b'?'Offline':'Ok'}})) + if(op==='GetStackActionState')return Object.fromEntries(['pulling','deploying','starting','restarting','pausing','unpausing','stopping','destroying'].map(k=>[k,false])) + if(op==='GetStack')return {...structuredClone(configs.get(p.stack)),info:{remote_contents:[{path:'compose.yml',contents:actualFiles.get(p.stack)}],remote_errors:fault.stale?['unavailable']:[],missing_files:[]}} + if(op==='ListStackServices')return containers.filter(c=>c.stack===p.stack).map(c=>({service:c.service,container:{id:c.Id}})) + if(op==='InspectContainer')return structuredClone(containers.find(c=>c.Id===p.container)) + if(op==='InspectImage')return {Id:'sha256:'+'f'.repeat(64),RepoDigests:[IMAGE.slice(0,-1)+'@'+d],Os:'linux',Architecture:fault.image?'arm64':'amd64',RootFS:{Type:'layers',Layers:[d]}} + throw Error('Unexpected read '+op) + }, + async refreshStackCache(stack){calls.push({op:'RefreshStackCache',stack})}, + async writeHostFile(stack,contents){calls.push({op:'WriteStackFileContents',stack});if(fault.unknownWrite)throw Object.assign(Error('private fixture never persist'),{uncertain:true});actualFiles.set(stack,contents)}, + async setHostReleaseConfig(stack,config){calls.push({op:'UpdateStack',stack});Object.assign(configs.get(stack).config,config);if(fault.configDrift)configs.get(stack).config.environment='changed'}, + async execute(op,p){ + calls.push({op,...p}) + if(op==='RunStackService'){ + assert.equal(p.no_deps,true);assert.equal(p.pull,false);assert.equal(p.service_ports,false);assert.equal(p.detach,false) + assert.equal(JSON.stringify(p).includes('migrate-bin'),false) + const marker=p.service==='postgres'?'STATE':'METADATA',value=p.service==='postgres'?state:target + return {status:'Complete',success:true,logs:[{stdout:'SCHEMA_UPGRADE_'+marker+'='+JSON.stringify(value)}]} + } + if(op==='PullStack'){assert.deepEqual(p.services,['db-migrate']);return} + const c=containers.find(c=>c.stack===p.stack&&c.service===p.services[0]) + if(op==='StopStack'){c.State.Running=false;c.State.Pid=0;return} + assert.equal(op,'DeployStack');assert.deepEqual(configs.get(p.stack).config.extra_args,RELEASE_ARGS);assert.equal(configs.get(p.stack).config.auto_pull,false) + c.Id='e'+c.Id.slice(1);c.Config.Image=IMAGE+b;c.Config.Env[0]='LINGXILOOP_COMMIT_SHA='+b;c.State.Running=true;c.State.Pid=2 + if(fault.containerImage)c.Image='sha256:'+'0'.repeat(64) + }, + } + const onProgress=async e=>events.push(e),ops=createHostReleaseOperations({plan,files,transport:t,onProgress,wait:async()=>{}}) + return {plan,ops,calls,events,configs,actualFiles,files,run:()=>runHostRelease({plan,ops,onProgress})} +} +test('real adapter/engine contract stages public host files, preserves core and full secret config, no pull/migration',async()=>{ + const f=fixture();await f.run() + assert.deepEqual(f.calls.filter(c=>c.op==='WriteStackFileContents').map(c=>c.stack),APPS) + assert.equal(f.actualFiles.get(CORE),f.files.get(CORE).source) + assert.equal(f.calls.some(c=>c.op==='PullStack'),false) + assert.equal(JSON.stringify(f.events).includes('private fixture'),false) + assert.ok(APPS.every(s=>f.configs.get(s).config.environment==='private fixture never persist')) +}) +test('host errors, platform mismatch and unexpected full-config change fail before stops',async()=>{ + for(const fault of [{hostDown:true},{stale:true},{image:true},{configDrift:true},{unknownWrite:true},{sourceImageMismatch:true}]){ + const f=fixture(fault);await assert.rejects(f.run()) + assert.equal(f.calls.some(c=>c.op==='StopStack'||c.op==='DeployStack'),false) + if(fault.unknownWrite)assert.equal(f.calls.at(-1).op,'WriteStackFileContents') + if(fault.configDrift){assert.equal(f.events.at(-1).status,'uncertain');assert.equal(f.calls.at(-1).op,'GetStack')} + if(fault.sourceImageMismatch)assert.equal(f.calls.some(c=>c.op==='WriteStackFileContents'),false) + } +}) +test('actual container image must equal verified host ID, not merely declared source',async()=>{ + const f=fixture({containerImage:true});await assert.rejects(f.run());assert.equal(f.calls.filter(c=>c.op==='DeployStack').length,1) +}) +test('explicit online preparation uses pull template then restores exact original files and config',async()=>{ + const f=fixture(),before=structuredClone(f.configs) + await runImagePreparation({plan:f.plan,ops:f.ops}) + assert.deepEqual(f.configs,before) + assert.deepEqual(f.calls.filter(c=>c.op==='PullStack').map(c=>c.stack),APPS) + assert.equal(f.calls.some(c=>['StopStack','DeployStack','UpdateStack'].includes(c.op)),false) + for(const stack of [...APPS,CORE])assert.equal(f.actualFiles.get(stack),f.files.get(stack).source) +}) diff --git a/scripts/host-release-transport.test.mjs b/scripts/host-release-transport.test.mjs new file mode 100644 index 000000000..b7a1c41f3 --- /dev/null +++ b/scripts/host-release-transport.test.mjs @@ -0,0 +1,35 @@ +import test from 'node:test' +import assert from 'node:assert/strict' +import {createTransport} from './schema-upgrade-komodo.mjs' +const sid='a'.repeat(24),id='b'.repeat(24),stack='lingxiloop-app-a',commit='c'.repeat(40) +function fixture(fault){ + const calls=[],events=[] + const transport=createTransport({apiKey:'fixture',apiSecret:'fixture',wait:async()=>{},onProgress:async e=>events.push(e),fetcher:async(url,options)=>{ + calls.push({url,params:JSON.parse(options.body)}) + if(url.endsWith('GetStack'))return Response.json({_id:{$oid:sid}}) + if(fault==='unknown')throw Error('secret fixture never emitted') + if(url.endsWith('UpdateStack'))return Response.json({}) + return Response.json({_id:{$oid:id},operation:fault==='wrong-operation'?'DeployStack':'WriteStackContents',target:{type:'Stack',id:fault==='wrong-target'?'d'.repeat(24):sid},status:'Complete',success:fault!=='failure'}) + }}) + return {transport,calls,events} +} +test('public host file write binds real WriteStackContents receipt and re-reads exact update',async()=>{ + const f=fixture();await f.transport.writeHostFile(stack,'services: {}') + assert.deepEqual(f.calls.map(x=>x.url.split('/').at(-1)),['GetStack','WriteStackFileContents','GetUpdate']) + assert.deepEqual(f.calls[1].params,{stack,file_path:'compose.yml',contents:'services: {}'}) +}) +test('wrong receipt/unknown write is never retried or treated as success',async()=>{ + for(const fault of ['unknown','wrong-operation','wrong-target','failure']){ + const f=fixture(fault);await assert.rejects(f.transport.writeHostFile(stack,'services: {}'),e=>!e.message.includes('secret fixture')) + assert.equal(f.calls.filter(x=>x.url.endsWith('WriteStackFileContents')).length,1) + assert.equal(f.events.some(x=>x.status==='complete'),false) + } +}) +test('native config patch changes only exact app provenance and cache/dependency policy',async()=>{ + const f=fixture();await f.transport.setHostReleaseConfig(stack,{commit,branch:'release/target',auto_pull:false,extra_args:['--no-deps','--pull','never']}) + assert.deepEqual(f.calls[1].params,{id:sid,config:{commit,branch:'release/target',auto_pull:false,extra_args:['--no-deps','--pull','never']}}) + for(const config of [{commit,branch:'release/target',auto_pull:true,extra_args:[]},{commit,branch:'release/target',auto_pull:false,extra_args:['--privileged']},{commit,branch:'release/target',auto_pull:false,extra_args:['--no-deps','--pull','never'],environment:'secret'}]){ + const a=fixture();await assert.rejects(a.transport.setHostReleaseConfig(stack,config));assert.equal(a.calls.length,0) + } + const a=fixture();await assert.rejects(a.transport.writeHostFile('lingxiloop-core-state','services: {}'));assert.equal(a.calls.length,0) +}) diff --git a/scripts/host-release.local.test.mjs b/scripts/host-release.local.test.mjs new file mode 100644 index 000000000..9ad7e927b --- /dev/null +++ b/scripts/host-release.local.test.mjs @@ -0,0 +1,57 @@ +// Real local Docker contract. No production destination, no registry pull. +import test from 'node:test' +import assert from 'node:assert/strict' +import {mkdtemp,writeFile,readFile,rm} from 'node:fs/promises' +import {tmpdir} from 'node:os' +import {join} from 'node:path' +import {execFileSync,spawnSync} from 'node:child_process' +import {assertHostComposeChange} from './host-release-compose.mjs' + +test('effective candidate preserves all existing policy, with only immutable image and never pull changes',async()=>{ + for(const side of ['a','b']){ + const original=await readFile(new URL(`../deploy/komodo/lingxiloop-app-${side}/compose.yml`,import.meta.url),'utf8') + const old=/lingxiloop-server:([a-f0-9]{40})/.exec(original)[1],next='e'.repeat(40) + const target=original.replaceAll(old,next).replace('pull_policy: always','pull_policy: never') + assert.equal(assertHostComposeChange(original,target,old,next),true) + assert.throws(()=>assertHostComposeChange(original,target.replace('read_only: true','read_only: false'),old,next)) + assert.throws(()=>assertHostComposeChange(original,target.replace('NODE_ENV: production','NODE_ENV: other'),old,next)) + assert.throws(()=>assertHostComposeChange(original,target.replace('lingxios-homes:/var/lib/lingxios/homes','different:/var/lib/lingxios/homes'),old,next)) + assert.throws(()=>assertHostComposeChange(original,target.replaceAll('pull_policy: never','pull_policy: always'),old,next)) + } +}) +test('real Compose up exact Komodo arguments override always, skip dependencies and never pull missing images',async()=>{ + const directory=await mkdtemp(join(tmpdir(),'host-release-compose-')),project='hostrelease'+process.pid,tag=`127.0.0.1:1/host-release-contract:${process.pid}` + const compose=join(directory,'compose.yml'),receipts=[] + const docker=(args,expected=0)=>{ + const r=spawnSync('docker',args,{encoding:'utf8',timeout:30000,maxBuffer:1024*1024}) + receipts.push({args,status:r.status}) + assert.equal(r.error,undefined);assert.equal(r.status,expected,`${args.join(' ')}: ${r.stderr}`);return r.stdout + } + const args=['compose','-p',project,'-f',compose] + try{ + // Existing local test image only; absence fails instead of silently pulling. + docker(['image','inspect','redis:7-alpine','--format','{{.Id}}']) + docker(['tag','redis:7-alpine',tag]) + await writeFile(compose,`services:\n db-migrate:\n image: ${tag}\n pull_policy: always\n command: ["sh", "-c", "sleep 120"]\n network_mode: none\n web:\n image: ${tag}\n pull_policy: always\n command: ["sh", "-c", "sleep 120"]\n network_mode: none\n depends_on: [db-migrate]\n`) + docker([...args,'up','-d','--no-deps','--pull','never','--','web']) + assert.equal(docker([...args,'ps','-a','--services']).trim(),'web') + // Native RunStackService supplies --rm/--no-tty/--no-deps, but no --pull. + await writeFile(compose,(await readFile(compose,'utf8')).replaceAll('pull_policy: always','pull_policy: never')) + const help=docker(['compose','run','--help']),noTTY=help.includes('--no-tty')?'--no-tty':'-T' + // Compose2.40.3 spells this --no-TTY/-T; Komodo2.3.3 emits --no-tty, + // supported by the production installation but not this older local CLI. + // Record the gap rather than claiming exact native oneoff command coverage. + receipts.push({operation:'oneoff-tty-compatibility',nativeLongFlagSupported:noTTY==='--no-tty',localFlag:noTTY}) + docker([...args,'run','--rm',noTTY,'--no-deps','--','db-migrate','true']) + assert.equal(docker([...args,'ps','-a','--services']).trim(),'web') + await writeFile(compose,(await readFile(compose,'utf8')).replaceAll(tag,tag+'missing')) + const missing=spawnSync('docker',[...args,'up','-d','--no-deps','--pull','never','--','web'],{encoding:'utf8',timeout:30000}) + assert.notEqual(missing.status,0);assert.doesNotMatch(missing.stderr,/Pulling|connection refused|Get "http/) + receipts.push({operation:'missing-cache-refused',status:missing.status}) + await writeFile(join(tmpdir(),'host-release-compose-contract.json'),JSON.stringify({passed:true,composeVersion:execFileSync('docker',['compose','version'],{encoding:'utf8'}).trim(),receipts},null,2)+'\n') + }finally{ + spawnSync('docker',[...args,'down','--remove-orphans'],{encoding:'utf8',timeout:30000}) + spawnSync('docker',['image','rm',tag],{encoding:'utf8',timeout:30000}) + await rm(directory,{recursive:true,force:true}) + } +}) diff --git a/scripts/host-release.mjs b/scripts/host-release.mjs new file mode 100644 index 000000000..e02afa4df --- /dev/null +++ b/scripts/host-release.mjs @@ -0,0 +1,201 @@ +// Explicit same-schema application release. No network work on import. +import {createHash} from 'node:crypto' +import {readFile,writeFile} from 'node:fs/promises' +import {execFile} from 'node:child_process' +import {promisify,isDeepStrictEqual as equal} from 'node:util' +import {isAbsolute,resolve} from 'node:path' +import {fileURLToPath,pathToFileURL} from 'node:url' +import {releaseDescriptor,assertInstalledTarget} from './schema-upgrade-maintenance.mjs' +import {assertHostComposeChange} from './host-release-compose.mjs' + +export const APPS=['lingxiloop-app-a','lingxiloop-app-b'],CORE='lingxiloop-core-state' +export const RELEASE_ARGS=['--no-deps','--pull','never'] +export const IMAGE='accel.way2api.fun/ghcr.io/lingxi-org/lingxiloop-server:' +export const check=(ok,message)=>{if(!ok)throw Error(message)} +export const hash=bytes=>createHash('sha256').update(bytes).digest('hex') +const sha=/^[a-f0-9]{40}$/,hex=/^[a-f0-9]{64}$/,digest=/^sha256:[a-f0-9]{64}$/ +const formalKeys=['runtime','schema','protocol','schemaHash','integrity','tarball','productVersion','migrations'] +export const metadataKeys=['sha',...formalKeys] +const select=(v,keys)=>Object.fromEntries(keys.map(k=>[k,v?.[k]])) +const stable=value=>JSON.stringify(value,(_,v)=>v&&typeof v==='object'&&!Array.isArray(v)?Object.fromEntries(Object.keys(v).sort().map(k=>[k,v[k]])):v) +export function sealHostPlan(value){const {planHash:_,...body}=value;return {...body,planHash:hash(stable(body))}} +const exactKeys=(v,keys)=>v&&equal(Object.keys(v).sort(),keys.toSorted()) +export function assertSameSchema(source,target){ + check(equal(select(source,formalKeys),select(target,formalKeys)),'Formal package/schema/inventory differs; dedicated migration required') + check(source.schema===13&&source.productVersion===48&&Array.isArray(source.migrations)&&source.migrations.length===48 + &&source.migrations.every((m,i)=>exactKeys(m,['version','name','checksum'])&&m.version===i+1&&/^[a-z0-9_]+$/.test(m.name??'')&&hex.test(m.checksum??'')),'Exact native13/product48 inventory required') +} +export function validateHostPlanShape(plan){ + check(exactKeys(plan,['format','createdAt','source','target','sourceAcceptanceSha256','qualification','stacks','planHash']) + &&plan.format==='same-schema-host-release/v1'&&equal(plan,sealHostPlan(plan))&&Number.isFinite(Date.parse(plan.createdAt)),'Invalid sealed host-release plan') + assertSameSchema(plan.source,plan.target) + for(const d of [plan.source,plan.target]){ + check(exactKeys(d,[...metadataKeys,'image'])&&sha.test(d.sha??'')&&/^\d+\.\d+\.\d+$/.test(d.runtime??'') + &&Number.isSafeInteger(d.protocol)&&d.protocol>0&&hex.test(d.schemaHash??'') + &&/^sha512-[\w+/]+={0,2}$/.test(d.integrity??'')&&d.tarball?.startsWith(`https://npm.pkg.github.com/download/@lyyzka/lingxios/${d.runtime}/`),'Invalid formal descriptor') + const i=d.image + check(exactKeys(i,['indexDigest','configDigest','manifestDigest','diffIds','auditSha256'])&&[i.indexDigest,i.configDigest,i.manifestDigest].every(x=>digest.test(x??'')) + &&hex.test(i.auditSha256??'')&&Array.isArray(i.diffIds)&&i.diffIds.length>0&&i.diffIds.every(x=>digest.test(x)),'Exact audited image descriptor required') + } + check(hex.test(plan.sourceAcceptanceSha256??'')&&exactKeys(plan.qualification,['commit','conclusion','ciUrl']) + &&plan.qualification.commit===plan.target.sha&&plan.qualification.conclusion==='success' + &&/^https:\/\/github\.com\/LingXi-Org\/LingxiLoop\/actions\/runs\/\d+$/.test(plan.qualification.ciUrl??''),'Accepted source and exact-target CI required') + check(Array.isArray(plan.stacks)&&plan.stacks.length===3,'Exactly two app sources and unchanged core required') + for(const [i,row] of plan.stacks.entries()){ + check(exactKeys(row,i===2?['stack','server','source','target']:['stack','server','source','target','preparation']) + &&row.stack===[...APPS,CORE][i]&&row.server===(i===1?'shanghai-b':'shanghai-a'),'Unexpected stack or server') + for(const ref of [row.source,row.target]){ + check(exactKeys(ref,['commit','branch','sha256','autoPull','extraArgs'])&&sha.test(ref.commit??'')&&hex.test(ref.sha256??'') + &&/^release\/[a-z0-9][a-z0-9._-]{0,95}$/.test(ref.branch??'')&&!ref.branch.includes('..')&&!ref.branch.endsWith('.')&&!ref.branch.endsWith('.lock') + &&typeof ref.autoPull==='boolean'&&(equal(ref.extraArgs,[])||equal(ref.extraArgs,RELEASE_ARGS)),'Invalid stack source policy') + } + if(i===2)check(equal(row.source,row.target)&&equal(row.source.extraArgs,[]),'Core source/policy must remain unchanged') + else{ + check(row.target.autoPull===false&&equal(row.target.extraArgs,RELEASE_ARGS),'App deployment must disable pull/dependencies') + check(exactKeys(row.preparation,['commit','sha256'])&&sha.test(row.preparation.commit??'')&&hex.test(row.preparation.sha256??''),'Explicit committed online pull template required') + } + } +} +export function assertWorkerBarrier(state){ + check(state?.readOnly===true&&state.fullVisibility===true,'Worker lease visibility unavailable') + for(const key of ['liveWork','liveSession','liveAttempts','liveCapture','liveOutbox','prepared'])check(state[key]===0,`Worker barrier not empty: ${key}`) +} +export function assertCompatibleState(state,descriptor){ + check(state?.readOnly===true&&state.fullVisibility===true&&state.schema===descriptor.schema&&state.installedSchema===descriptor.schema + &&state.runtime===descriptor.runtime&&state.protocol===descriptor.protocol&&state.schemaHash===descriptor.schemaHash + &&state.product===48&&state.count===48&&equal(state.migrations,descriptor.migrations),'Actual database provenance changed') +} +export function assertImageIdentity(actual,audit){ + check(digest.test(actual?.Id??'')&&actual.RepoDigests?.some(x=>x===IMAGE.slice(0,-1)+'@'+audit.indexDigest) + &&actual.Os==='linux'&&actual.Architecture==='amd64'&&actual.RootFS?.Type==='layers'&&equal(actual.RootFS.Layers,audit.diffIds),'Cached image does not match audited index/platform/layers') + if(actual.Descriptor)check(actual.Descriptor.digest===audit.indexDigest||actual.Descriptor.digest===audit.manifestDigest,'Image descriptor differs from qualified index/platform') + return actual.Id +} +function assertInventory(rows,plan,original,stopped=new Set(),deployed=new Map()){ + check(Array.isArray(rows)&&rows.length===4&&new Set(rows.map(x=>x.id)).size===4,'Four distinct application containers required') + for(const stack of APPS)for(const service of ['lingxiloop','worker']){ + const key=stack+':'+service,matches=rows.filter(x=>x.stack===stack&&x.service===service),r=matches[0],old=original?.find(x=>x.stack===stack&&x.service===service) + check(matches.length===1&&hex.test(r.id??'')&&r.source===(deployed.has(key)?plan.target.sha:plan.source.sha)&&r.restarting===false + &&r.restartCount===0&&digest.test(r.imageId??''),'Container identity/state differs from release') + if(old)check(equal(r.policy,old.policy),'Container security policy changed') + if(!deployed.has(key)&&old)check(r.id===old.id&&r.imageId===old.imageId,'Original container replaced outside ordered release') + if(deployed.has(key))check(r.id===deployed.get(key).id&&r.imageId===deployed.get(key).imageId,'Healthy target container replaced outside ordered release') + if(stopped.has(key)&&!deployed.has(key))check(r.running===false&&r.pid===0,'Old Worker still running') + else check(r.running===true&&r.healthy===true&&r.pid>0,'Expected application service unhealthy') + } +} +async function failed({error,ops,plan,record}){ + let recovery='unknown' + if(!error?.uncertain)try{assertCompatibleState(await ops.state(),plan.source);recovery='same-schema-recovery-requires-review'}catch{} + await record({phase:'stopped',status:error?.uncertain?'uncertain':'failed',recovery,noAutomaticRetry:true}).catch(()=>{}) + throw Object.assign(Error('Host release stopped; resolve the private receipt before any further action'),{uncertain:error?.uncertain===true}) +} +export async function runHostRelease({plan,ops,onProgress=async()=>{}}){ + validateHostPlanShape(plan) + const record=e=>onProgress({planHash:plan.planHash,...e}) + try{ + await ops.assertIdle();const original=await ops.inspect();assertInventory(original,plan) + assertCompatibleState(await ops.state(),plan.source) + await ops.verifyImages('source');await ops.verifyImages('target') + assertInventory(await ops.inspect(),plan,original) + await record({phase:'preflight',status:'verified',original}) + for(const stack of APPS){await ops.assertIdle();await ops.stage(stack);assertInventory(await ops.inspect(),plan,original)} + await ops.verifyImages('target') + for(const stack of APPS)check(equal(select(await ops.metadata(stack),metadataKeys),select(plan.target,metadataKeys)),'Actual target metadata mismatch') + assertCompatibleState(await ops.state(),plan.source);assertInventory(await ops.inspect(),plan,original) + await record({phase:'online',status:'prepared'}) + const stopped=new Set(),deployed=new Map() + for(const stack of APPS){await ops.assertIdle();await ops.stop(stack,'worker');stopped.add(stack+':worker');assertInventory(await ops.inspect(),plan,original,stopped);await record({phase:'drain',status:'stopped',stack,service:'worker'})} + const barrier=async()=>{assertInventory(await ops.inspect(),plan,original,stopped,deployed);const state=await ops.state();assertCompatibleState(state,plan.source);assertWorkerBarrier(state);await record({phase:'drain',status:'zero-leases',state})} + await barrier();await ops.wait(2000);await barrier() + for(const service of ['lingxiloop','worker'])for(const stack of APPS){ + await ops.assertIdle();await ops.verifyImages('target') + assertCompatibleState(await ops.state(),plan.target) + // Before the first new Worker, every native lease must still be empty. + if(![...deployed.keys()].some(k=>k.endsWith(':worker')))await barrier() + assertInventory(await ops.inspect(),plan,original,stopped,deployed) + await ops.deploy(stack,service);await ops.verifyHealthy(stack,service) + const rows=await ops.inspect(),healthy=rows.find(r=>r.stack===stack&&r.service===service) + check(healthy,'Healthy target identity unavailable');deployed.set(stack+':'+service,{id:healthy.id,imageId:healthy.imageId}) + assertInventory(rows,plan,original,stopped,deployed) + assertCompatibleState(await ops.state(),plan.target) + await record({phase:'rollout',status:'healthy',stack,service,containerId:healthy.id,imageId:healthy.imageId}) + } + await ops.assertIdle();await ops.verifyImages('target');const state=await ops.state();assertCompatibleState(state,plan.target) + const containers=await ops.inspect();assertInventory(containers,plan,original,stopped,deployed) + const result={phase:'complete',status:'complete',planHash:plan.planHash,state,containers} + await record(result);return result + }catch(error){return failed({error,ops,plan,record})} +} +export async function runImagePreparation({plan,ops,onProgress=async()=>{}}){ + validateHostPlanShape(plan);const record=e=>onProgress({planHash:plan.planHash,...e}) + try{ + await ops.assertIdle();const original=await ops.inspect();assertInventory(original,plan) + assertCompatibleState(await ops.state(),plan.source);await ops.verifyImages('source') + assertInventory(await ops.inspect(),plan,original) + for(const stack of APPS){ + await ops.assertIdle();await ops.stagePreparation(stack);assertInventory(await ops.inspect(),plan,original) + await ops.pullPrepared(stack);await ops.verifyImages('target',stack);assertInventory(await ops.inspect(),plan,original) + await ops.restoreSource(stack);assertInventory(await ops.inspect(),plan,original) + await record({phase:'image-preparation',status:'restored-source',stack}) + } + await ops.assertIdle();await ops.verifyImages('target');assertCompatibleState(await ops.state(),plan.source) + const result={phase:'complete',status:'complete',planHash:plan.planHash,mode:'online-images-only',original} + await record(result);return result + }catch(error){return failed({error,ops,plan,record})} +} + +const root=fileURLToPath(new URL('../',import.meta.url)),run=promisify(execFile) +async function git(args){return (await run('git',args,{cwd:root,encoding:'utf8',maxBuffer:2*1024*1024})).stdout} +export async function validateHostPlan(plan,{acceptance,sourceAudit,targetAudit}){ + validateHostPlanShape(plan) + check(hash(acceptance)===plan.sourceAcceptanceSha256,'Source acceptance bytes changed') + const accepted=JSON.parse(acceptance) + check(accepted.passed===true&&accepted.candidate===plan.source.sha&&Object.keys(accepted.checks??{}).length>0&&Object.values(accepted.checks).every(x=>x===true),'Source has no successful sealed acceptance') + for(const [descriptor,bytes] of [[plan.source,sourceAudit],[plan.target,targetAudit]]){ + const audit=JSON.parse(bytes),i=descriptor.image + check(hash(bytes)===i.auditSha256&&audit.passed===true&&audit.sourceCommit===descriptor.sha&&audit.indexDigest===i.indexDigest + &&audit.configDigest===i.configDigest&&audit.linuxAmd64ManifestDigest===i.manifestDigest&&equal(audit.layers?.map(x=>x.diffId),i.diffIds),'Image audit bytes or provenance mismatch') + const derived=await releaseDescriptor(descriptor.sha,descriptor) + check(equal(derived,select(descriptor,metadataKeys)),'Committed package/inventory differs from descriptor') + } + await assertInstalledTarget(plan.target) + const files=new Map() + for(const row of plan.stacks){ + const values={} + for(const which of ['source','target',...(row.preparation?['preparation']:[])]){ + const ref=row[which],path=`deploy/komodo/${row.stack}/compose.yml`,entry=await git(['ls-tree',ref.commit,path]) + check(new RegExp('^100644 blob [a-f0-9]{40}\\t'+path+'\\n$').test(entry),'Source must be a tracked regular public Compose file') + const content=await git(['show',`${ref.commit}:${path}`]);check(hash(content)===ref.sha256,'Committed Compose hash differs') + values[which]=content + } + if(row.stack!==CORE){ + assertHostComposeChange(values.source,values.target,plan.source.sha,plan.target.sha,'never') + assertHostComposeChange(values.target,values.preparation,plan.target.sha,plan.target.sha,'always') + check((await git(['show','-s','--format=%B',row.target.commit])).includes('[skip deploy]'),'Candidate manifests must suppress legacy rollout') + }else check(values.source===values.target,'Core template changed') + files.set(row.stack,values) + } + return files +} +async function main(){ + const args=process.argv.slice(2),values={},allowed=['--execute','--prepare-images','--input','--output','--acceptance','--source-audit','--target-audit'] + if(args.length===1&&args[0]==='--help'){console.log('Offline by default. --input --output --acceptance --source-audit --target-audit . All paths absolute. --execute performs same-schema release; --prepare-images explicitly prepares images online and restores source files. No automatic retry/recovery.');return} + for(let i=0;iabort.abort(),out={planHash:plan.planHash,mode:values['--prepare-images']?'online-images-only':'same-schema-release',startedAt:new Date().toISOString(),complete:false,events:[]} + await writeFile(values['--output'],JSON.stringify(out)+'\n',{flag:'wx',mode:0o600}) + const onProgress=async e=>{out.events.push({...e,observedAt:new Date().toISOString()});if(e.phase==='complete')out.complete=true;await writeFile(values['--output'],JSON.stringify(out,null,2)+'\n',{mode:0o600})} + process.once('SIGINT',stop);process.once('SIGTERM',stop) + try{ + const {createHostReleaseOperations}=await import('./host-release-komodo.mjs') + const ops=createHostReleaseOperations({plan,files,apiKey:process.env.KOMODO_API_KEY,apiSecret:process.env.KOMODO_API_SECRET,signal:abort.signal,onProgress}) + await (values['--prepare-images']?runImagePreparation:runHostRelease)({plan,ops,onProgress}) + console.log(JSON.stringify({status:'complete',mode:out.mode,planHash:plan.planHash})) + }finally{process.off('SIGINT',stop);process.off('SIGTERM',stop)} +} +if(process.argv[1]&&pathToFileURL(resolve(process.argv[1])).href===import.meta.url)main().catch(()=>{console.error('Host release failed; inspect the private receipt. No automatic retry/recovery.');process.exitCode=1}) diff --git a/scripts/host-release.test.mjs b/scripts/host-release.test.mjs new file mode 100644 index 000000000..327924e92 --- /dev/null +++ b/scripts/host-release.test.mjs @@ -0,0 +1,88 @@ +import test from 'node:test' +import assert from 'node:assert/strict' +import {sealHostPlan,assertSameSchema,assertWorkerBarrier,assertImageIdentity,validateHostPlanShape,runHostRelease,runImagePreparation} from './host-release.mjs' + +const sha='a'.repeat(40),next='b'.repeat(40),hash='c'.repeat(64),digest='sha256:'+hash +const migrations=Array.from({length:48},(_,i)=>({version:i+1,name:'migration_'+(i+1),checksum:hash})) +const formal={runtime:'3.5.8',schema:13,protocol:11,schemaHash:hash,integrity:'sha512-'+Buffer.alloc(64).toString('base64'),tarball:'https://npm.pkg.github.com/download/@lyyzka/lingxios/3.5.8/fixture',productVersion:48,migrations} +const image={indexDigest:digest,configDigest:'sha256:'+'d'.repeat(64),manifestDigest:'sha256:'+'e'.repeat(64),diffIds:[digest],auditSha256:hash} +const source={sha,...formal,image},target={...source,sha:next} +const apps=['lingxiloop-app-a','lingxiloop-app-b'],core='lingxiloop-core-state' +const sourceRef={commit:sha,branch:'release/source',sha256:hash,autoPull:true,extraArgs:[]} +const targetRef={commit:next,branch:'release/target',sha256:'d'.repeat(64),autoPull:false,extraArgs:['--no-deps','--pull','never']} +const plan=sealHostPlan({format:'same-schema-host-release/v1',createdAt:'2026-10-20T00:00:00Z',source,target,sourceAcceptanceSha256:hash,qualification:{commit:next,conclusion:'success',ciUrl:'https://github.com/LingXi-Org/LingxiLoop/actions/runs/123'},stacks:[...apps,core].map((stack,i)=>({stack,server:i===1?'shanghai-b':'shanghai-a',source:sourceRef,target:i===2?sourceRef:targetRef,...(i!==2?{preparation:{commit:next,sha256:'e'.repeat(64)}}:{})}))}) +function state(){return {...formal,installedSchema:13,product:48,count:48,readOnly:true,fullVisibility:true,liveWork:0,liveSession:0,liveAttempts:0,liveCapture:0,liveOutbox:0,prepared:0,otherClients:20,queued:12,pendingCapture:4}} +function fixture(fault={}){ + const trace=[],events=[],rows=apps.flatMap((stack,i)=>['lingxiloop','worker'].map((service,j)=>({stack,server:i?'shanghai-b':'shanghai-a',service,id:String(i*2+j+1).repeat(64),source:sha,imageId:digest,running:true,restarting:false,restartCount:0,pid:1,healthy:true,policy:{readOnly:service==='worker'}}))) + let staged=0,deployed=0,stops=0,targetInspections=0 + const ops={ + async assertIdle(){trace.push('idle');if(fault.idle)throw Error('idle')}, + async inspect(){trace.push('inspect');if(fault.replacedTarget&&deployed===1&&++targetInspections===2)rows[0].id='e'+rows[0].id.slice(1);return structuredClone(rows)}, + async state(){trace.push('state');const s=state();if(fault.busy&&stops===2)s.liveWork=1;if(fault.schema&&deployed)s.schemaHash='e'.repeat(64);return s}, + async verifyImages(which='target'){trace.push('images:'+which);if(fault.image&&which==='target')throw Error('missing image')}, + async stage(stack){trace.push('stage:'+stack);staged++;if(fault.stage&&staged===2)throw Object.assign(Error('unknown'),{uncertain:true})}, + async metadata(stack){trace.push('metadata:'+stack);return {...target,...(fault.metadata&&stack===apps[1]?{schema:14}:{})}}, + async stop(stack,service){trace.push('stop:'+stack+':'+service);stops++;if(fault.stopUnknown)throw Object.assign(Error('unknown'),{uncertain:true});const row=rows.find(x=>x.stack===stack&&x.service===service);if(!fault.stillRunning){row.running=false;row.pid=0}}, + async deploy(stack,service){trace.push('deploy:'+stack+':'+service);deployed++;if(fault.deployUnknown)throw Object.assign(Error('unknown'),{uncertain:true});const row=rows.find(x=>x.stack===stack&&x.service===service);Object.assign(row,{source:next,id:'f'+row.id.slice(1),running:true,pid:2});if(fault.policy)row.policy.readOnly=!row.policy.readOnly}, + async verifyHealthy(stack,service){trace.push('health:'+stack+':'+service);if(fault.health)throw Error('unhealthy')}, + async wait(){trace.push('wait')}, + async stagePreparation(stack){trace.push('prepare:'+stack)}, + async pullPrepared(stack){trace.push('pull:'+stack);if(fault.pullUnknown)throw Object.assign(Error('unknown'),{uncertain:true})}, + async restoreSource(stack){trace.push('restore:'+stack)}, + } + return {ops,trace,events,run:()=>runHostRelease({plan,ops,onProgress:async e=>events.push(e)}),prepare:()=>runImagePreparation({plan,ops,onProgress:async e=>events.push(e)})} +} +test('same formal package and all48 checksums required; native13 alone is insufficient',()=>{ + assert.doesNotThrow(()=>assertSameSchema(source,target)) + for(const key of ['runtime','schema','protocol','schemaHash','integrity','tarball','productVersion','migrations']){ + const bad=structuredClone(target);bad[key]=key==='migrations'?migrations.slice(1):typeof bad[key]==='number'?bad[key]+1:'different' + assert.throws(()=>assertSameSchema(source,bad)) + } +}) +test('distinct format and sealed per-stack core identity; exact app policy only',()=>{ + assert.doesNotThrow(()=>validateHostPlanShape(plan)) + for(const mutate of [p=>p.format=2,p=>p.stacks[2].target.commit=next,p=>p.stacks[0].target.extraArgs=['--no-deps'],p=>p.stacks[0].target.autoPull=true,p=>p.target.migrations[1].checksum='d'.repeat(64)]){ + const bad=JSON.parse(JSON.stringify(plan));mutate(bad);assert.throws(()=>validateHostPlanShape(sealHostPlan(bad))) + } + const bad=structuredClone(plan);bad.createdAt='later';assert.throws(()=>validateHostPlanShape(bad)) +}) +test('worker barrier allows Web clients and queue but rejects every live lease and prepared transaction',()=>{ + assert.doesNotThrow(()=>assertWorkerBarrier(state())) + for(const key of ['liveWork','liveSession','liveAttempts','liveCapture','liveOutbox','prepared'])assert.throws(()=>assertWorkerBarrier({...state(),[key]:1})) + for(const key of ['readOnly','fullVisibility'])assert.throws(()=>assertWorkerBarrier({...state(),[key]:false})) +}) +test('actual per-host image ID is not inferred from config digest',()=>{ + const actual={Id:'sha256:'+'f'.repeat(64),RepoDigests:['accel.way2api.fun/ghcr.io/lingxi-org/lingxiloop-server@'+digest],Architecture:'amd64',Os:'linux',RootFS:{Type:'layers',Layers:[digest]}} + assert.equal(assertImageIdentity(actual,image),actual.Id) + for(const patch of [{Architecture:'arm64'},{RepoDigests:[]},{RootFS:{Type:'layers',Layers:[]}}])assert.throws(()=>assertImageIdentity({...actual,...patch},image)) +}) +test('online qualification precedes stop; only both Workers stop; WebA/B then WorkerA/B',async()=>{ + const f=fixture();const result=await f.run();assert.equal(result.status,'complete') + const stop=f.trace.findIndex(x=>x.startsWith('stop:')) + for(const stack of apps)assert.ok(f.trace.indexOf('metadata:'+stack)x.startsWith('stop:')),apps.map(s=>'stop:'+s+':worker')) + assert.deepEqual(f.trace.filter(x=>x.startsWith('deploy:')),['lingxiloop','worker'].flatMap(s=>apps.map(a=>'deploy:'+a+':'+s))) + assert.equal(f.trace.some(x=>/pull|migrate|backup|restore/.test(x)),false) +}) +test('missing image never stages; partial unknown stage never stops or makes further requests',async()=>{ + const a=fixture({image:true});await assert.rejects(a.run());assert.equal(a.trace.some(x=>x.startsWith('stage:')||x.startsWith('stop:')),false) + const b=fixture({stage:true});await assert.rejects(b.run());assert.equal(b.trace.some(x=>x.startsWith('stop:')),false);assert.equal(b.trace.at(-1),'stage:'+apps[1]);assert.equal(b.events.at(-1).status,'uncertain') +}) +test('metadata mismatch leaves all services running',async()=>{ + const f=fixture({metadata:true});await assert.rejects(f.run());assert.equal(f.trace.some(x=>x.startsWith('stop:')),false) +}) +test('live leases, ineffective stop and unknown stop prevent new Web',async()=>{ + for(const fault of [{busy:true},{stillRunning:true},{stopUnknown:true}]){const f=fixture(fault);await assert.rejects(f.run());assert.equal(f.trace.some(x=>x.startsWith('deploy:')),false)} +}) +test('unknown deploy stops immediately; health/policy/schema failure prevents remaining rollout',async()=>{ + for(const fault of [{deployUnknown:true},{health:true},{policy:true},{schema:true},{replacedTarget:true}]){ + const f=fixture(fault);await assert.rejects(f.run());assert.equal(f.trace.filter(x=>x.startsWith('deploy:')).length,1) + if(fault.deployUnknown)assert.equal(f.trace.at(-1),'deploy:'+apps[0]+':lingxiloop') + } +}) +test('explicit image preparation restores accepted host source and never stops/deploys',async()=>{ + const f=fixture();await f.prepare() + assert.deepEqual(f.trace.filter(x=>/^(prepare|pull|restore):/.test(x)),apps.flatMap(s=>['prepare:'+s,'pull:'+s,'restore:'+s])) + assert.equal(f.trace.some(x=>/^(stop|deploy|metadata):/.test(x)),false) + const bad=fixture({pullUnknown:true});await assert.rejects(bad.prepare());assert.equal(bad.trace.at(-1),'pull:'+apps[0]);assert.equal(bad.trace.some(x=>x.startsWith('restore:')),false) +}) diff --git a/scripts/schema-upgrade-komodo.mjs b/scripts/schema-upgrade-komodo.mjs index e1c67d60b..ee344eecc 100644 --- a/scripts/schema-upgrade-komodo.mjs +++ b/scripts/schema-upgrade-komodo.mjs @@ -82,7 +82,43 @@ export function createTransport({apiKey,apiSecret,signal,fetcher=fetch,wait=dela // Native cache refresh does not edit configuration or execute a service. await request('write','RefreshStackCache',{stack}) } - return {read,execute,refreshStackCache} + async function writeHostFile(stack,contents){ + check(APPS.includes(stack)&&typeof contents==='string'&&Buffer.byteLength(contents)<=512*1024,'Only public application Compose staging allowed') + const details=await read('GetStack',{stack}),stackId=details?._id?.$oid + check(ID.test(stackId??''),'Stack identity unavailable') + let id,submitted=false,terminal=false + try{ + await onProgress({phase:'komodo',status:'submitting',operation:'WriteStackFileContents',stack});signal?.throwIfAborted();submitted=true + const accepted=await request('write','WriteStackFileContents',{stack,file_path:'compose.yml',contents});id=accepted?._id?.$oid + check(ID.test(id??''),'Missing file-write update identity') + await onProgress({phase:'komodo',status:'pending',operation:'WriteStackFileContents',stack,updateId:id}) + const verify=u=>check(u?._id?.$oid===id&&u.operation==='WriteStackContents'&&u.target?.type==='Stack'&&u.target.id===stackId&&u.status==='Complete'&&typeof u.success==='boolean','File-write receipt identity mismatch') + // This native write is synchronous in v2.3.3; no speculative resubmission. + verify(accepted);terminal=true;check(accepted.success===true,'File write failed') + const confirmed=await read('GetUpdate',{id});verify(confirmed);check(confirmed.success===true,'File-write confirmation failed') + await onProgress({phase:'komodo',status:'complete',operation:'WriteStackFileContents',stack,updateId:id});return confirmed + }catch{ + const uncertain=submitted&&!terminal + await onProgress({phase:'komodo',status:uncertain?'uncertain':'failed',operation:'WriteStackFileContents',stack,...(ID.test(id??'')?{updateId:id}:{})}).catch(()=>{}) + throw Object.assign(Error('Host file write failed; resolve saved update before continuing'),{uncertain}) + } + } + async function setHostReleaseConfig(stack,config){ + check(APPS.includes(stack)&&config&&Object.keys(config).sort().join()==='auto_pull,branch,commit,extra_args' + &&HEX40.test(config.commit??'')&&/^release\/[a-z0-9][a-z0-9._-]{0,95}$/.test(config.branch??'')&&!config.branch.includes('..') + &&!config.branch.endsWith('.')&&!config.branch.endsWith('.lock')&&config.auto_pull===false + &&JSON.stringify(config.extra_args)==='["--no-deps","--pull","never"]','Only exact app host-release policy allowed') + const details=await read('GetStack',{stack}),id=details?._id?.$oid;check(ID.test(id??''),'Stack identity unavailable') + await onProgress({phase:'host-config',status:'submitting',stack,fields:Object.keys(config)}) + try{ + await request('write','UpdateStack',{id,config}) + await onProgress({phase:'host-config',status:'acknowledged',stack}) + } + catch{await onProgress({phase:'host-config',status:'uncertain',stack}).catch(()=>{});throw Object.assign(Error('Host configuration outcome uncertain; no retry'),{uncertain:true})} + // UpdateStack may return an empty body-shaped object. Caller verifies exact + // full config readback; neither HTTP success nor these fields imply success. + } + return {read,execute,refreshStackCache,writeHostFile,setHostReleaseConfig} } export function validateStateReceipt(value) { const numbers=['schema','installedSchema','protocol','count','product','liveWork','liveSession','liveAttempts','liveCapture','liveOutbox','otherClients','prepared','queued','pendingCapture'] @@ -120,7 +156,7 @@ SELECT 'SCHEMA_UPGRADE_STATE='||jsonb_build_object( 'pendingCapture',(SELECT count(*) FROM lingxios.agent_memory_capture WHERE completed_at IS NULL) )::text; COMMIT;` -const stateCommand=`set -eu; set +x; export PGPASSWORD="$POSTGRES_PASSWORD" PGCONNECT_TIMEOUT=10; exec timeout -k 5 30 psql -X -qAt -h postgres -U "$POSTGRES_USER" -d "$POSTGRES_DB" -v ON_ERROR_STOP=1 -c ${quote(STATE_SQL)}` +export const stateCommand=`set -eu; set +x; export PGPASSWORD="$POSTGRES_PASSWORD" PGCONNECT_TIMEOUT=10; exec timeout -k 5 30 psql -X -qAt -h postgres -U "$POSTGRES_USER" -d "$POSTGRES_DB" -v ON_ERROR_STOP=1 -c ${quote(STATE_SQL)}` export const METADATA_CODE=`import{readFile,readdir}from'node:fs/promises';import{createHash}from'node:crypto';import{releaseVersions,packageResources}from'@lyyzka/lingxios'; const hash=x=>createHash('sha256').update(x).digest('hex'); const lock=JSON.parse(await readFile('package-lock.json','utf8')).packages['node_modules/@lyyzka/lingxios']; From 731482a1fd98ed781bb1346273ee63fe92fef9f7 Mon Sep 17 00:00:00 2001 From: leecyang <164771957+leecyang@users.noreply.github.com> Date: Tue, 20 Oct 2026 14:54:57 +0000 Subject: [PATCH 3/3] fix: preserve uncertain host release classification [skip deploy] --- docs/host-release-failure-model.md | 2 +- scripts/host-release.mjs | 8 ++++---- scripts/host-release.test.mjs | 11 +++++++++-- 3 files changed, 14 insertions(+), 7 deletions(-) diff --git a/docs/host-release-failure-model.md b/docs/host-release-failure-model.md index 8049f8af9..77e60ed7b 100644 --- a/docs/host-release-failure-model.md +++ b/docs/host-release-failure-model.md @@ -14,7 +14,7 @@ This is an application-only release transaction, separate from native12→13 mai 8. Partial host write or config update, lost response, wrong update operation/target: record submitting/known update ID before progressing, require terminal success and exact readback. Never retry unknown outcomes, never overwrite the prior receipt, never deploy after a failed stage. 9. Worker stop accepted but still running, identity replaced, live native leases/attempts/capture/outbox or prepared transactions: fail before any new Web. Both old Workers stop first; require two fresh zero-lease observations. Web connections and queued work are allowed; this is not a database-wide no-client barrier. Establish image qualification and re-inspect the original containers before the first source write. Once a target service becomes healthy, retain its exact container identity; a later replacement with the same tag is not silently accepted. 10. Web can remain available while Workers drain. Update Web A, Web B, Worker A, Worker B, verifying exact image and actual health each time. Recheck unchanged schema throughout. After new Worker A begins, its legitimate leases must not be confused with old-worker leakage; all old Workers must remain stopped/replaced. -11. Target unhealthy or policy changes: stop the sequence. Unknown operation => no further network action; known failure => only a fresh read-only compatibility check for recovery classification, never automatic rollback. +11. Target unhealthy or policy changes: stop the sequence. Unknown operation => no further network action; known failure => only a fresh read-only compatibility check for recovery classification, never automatic rollback. If that classification oneoff itself has an unknown result, propagate uncertainty to the final receipt and caller; do not let the earlier known failure hide the new pending operation. 12. Recovery: preserve public source bytes/identities, original container evidence, stage and operation receipts. Only a separately reviewed recovery may restore prior public templates and redeploy the accepted compatible image after resolving every known update and proving the exact DB inventory again. No old-schema restore, no credential or environment export, no automatic replay of a partial receipt. 13. Two executors: exclusive local receipt creation and recorded ownership are necessary, but not a distributed lock. Native action/procedure idle gates are checked before every mutation. The designated sole production executor must serialize manual/CI entrypoints; this driver does not claim cross-host locking. 14. Receipt/log leakage: no config/environment, raw host source, API reply/log, credential, prompt or memory body enters progress/errors. Native WriteStackFileContents logs its argument, so send only exact tracked public Compose templates. diff --git a/scripts/host-release.mjs b/scripts/host-release.mjs index e02afa4df..e6a7ecd1e 100644 --- a/scripts/host-release.mjs +++ b/scripts/host-release.mjs @@ -85,10 +85,10 @@ function assertInventory(rows,plan,original,stopped=new Set(),deployed=new Map() } } async function failed({error,ops,plan,record}){ - let recovery='unknown' - if(!error?.uncertain)try{assertCompatibleState(await ops.state(),plan.source);recovery='same-schema-recovery-requires-review'}catch{} - await record({phase:'stopped',status:error?.uncertain?'uncertain':'failed',recovery,noAutomaticRetry:true}).catch(()=>{}) - throw Object.assign(Error('Host release stopped; resolve the private receipt before any further action'),{uncertain:error?.uncertain===true}) + let recovery='unknown',uncertain=error?.uncertain===true + if(!uncertain)try{assertCompatibleState(await ops.state(),plan.source);recovery='same-schema-recovery-requires-review'}catch(classificationError){uncertain=classificationError?.uncertain===true} + await record({phase:'stopped',status:uncertain?'uncertain':'failed',recovery,noAutomaticRetry:true}).catch(()=>{}) + throw Object.assign(Error('Host release stopped; resolve the private receipt before any further action'),{uncertain}) } export async function runHostRelease({plan,ops,onProgress=async()=>{}}){ validateHostPlanShape(plan) diff --git a/scripts/host-release.test.mjs b/scripts/host-release.test.mjs index 327924e92..46ea84cc2 100644 --- a/scripts/host-release.test.mjs +++ b/scripts/host-release.test.mjs @@ -14,11 +14,11 @@ const plan=sealHostPlan({format:'same-schema-host-release/v1',createdAt:'2026-10 function state(){return {...formal,installedSchema:13,product:48,count:48,readOnly:true,fullVisibility:true,liveWork:0,liveSession:0,liveAttempts:0,liveCapture:0,liveOutbox:0,prepared:0,otherClients:20,queued:12,pendingCapture:4}} function fixture(fault={}){ const trace=[],events=[],rows=apps.flatMap((stack,i)=>['lingxiloop','worker'].map((service,j)=>({stack,server:i?'shanghai-b':'shanghai-a',service,id:String(i*2+j+1).repeat(64),source:sha,imageId:digest,running:true,restarting:false,restartCount:0,pid:1,healthy:true,policy:{readOnly:service==='worker'}}))) - let staged=0,deployed=0,stops=0,targetInspections=0 + let staged=0,deployed=0,stops=0,targetInspections=0,stateReads=0 const ops={ async assertIdle(){trace.push('idle');if(fault.idle)throw Error('idle')}, async inspect(){trace.push('inspect');if(fault.replacedTarget&&deployed===1&&++targetInspections===2)rows[0].id='e'+rows[0].id.slice(1);return structuredClone(rows)}, - async state(){trace.push('state');const s=state();if(fault.busy&&stops===2)s.liveWork=1;if(fault.schema&&deployed)s.schemaHash='e'.repeat(64);return s}, + async state(){trace.push('state');if(++stateReads>fault.stateUnknownAfter)throw Object.assign(Error('unknown state'),{uncertain:true});const s=state();if(fault.busy&&stops===2)s.liveWork=1;if(fault.schema&&deployed)s.schemaHash='e'.repeat(64);return s}, async verifyImages(which='target'){trace.push('images:'+which);if(fault.image&&which==='target')throw Error('missing image')}, async stage(stack){trace.push('stage:'+stack);staged++;if(fault.stage&&staged===2)throw Object.assign(Error('unknown'),{uncertain:true})}, async metadata(stack){trace.push('metadata:'+stack);return {...target,...(fault.metadata&&stack===apps[1]?{schema:14}:{})}}, @@ -71,6 +71,13 @@ test('missing image never stages; partial unknown stage never stops or makes fur test('metadata mismatch leaves all services running',async()=>{ const f=fixture({metadata:true});await assert.rejects(f.run());assert.equal(f.trace.some(x=>x.startsWith('stop:')),false) }) +test('known failure followed by uncertain classification oneoff remains finally uncertain',async()=>{ + const f=fixture({metadata:true,stateUnknownAfter:1}) + await assert.rejects(f.run(),e=>e.uncertain===true) + assert.equal(f.events.at(-1).status,'uncertain');assert.equal(f.events.at(-1).recovery,'unknown') + assert.equal(f.trace.at(-1),'state');assert.equal(f.trace.filter(x=>x==='state').length,2) + assert.equal(f.trace.some(x=>x.startsWith('stop:')||x.startsWith('deploy:')),false) +}) test('live leases, ineffective stop and unknown stop prevent new Web',async()=>{ for(const fault of [{busy:true},{stillRunning:true},{stopUnknown:true}]){const f=fixture(fault);await assert.rejects(f.run());assert.equal(f.trace.some(x=>x.startsWith('deploy:')),false)} })