Skip to content

Speed up Labelle WASM builds: Docker CI image, compiler memory, linking and engine boundaries #528

New issue
New issue
Open
3 of 6 issues completed
Open
Speed up Labelle WASM builds: Docker CI image, compiler memory, linking and engine boundaries#528
3 of 6 issues completed

Description

@apotema
apotema
opened on Oct 30, 2026

Objective

Make Labelle WASM builds reproducible, faster and less memory-intensive, then adopt the released improvements in Flying Platform's GitLab CI. This ticket tracks the Docker builder image and all performance work below, as requested by the owner.

Coordinate implementation across labelle-cli, labelle-engine, labelle-assembler, labelle-web and labelle-bgfx. labelle-tools is archived and read-only. Audit existing Docker support before introducing another build path.

Measured baseline

Local Windows investigation on 2026-10-30, with one heavy job at a time and Zig builds limited to -j2:

Experiment Game compilation Peak private compiler memory
Original ReleaseSafe code 242 s 10.41 GiB
Debug information stripped 221 s 9.66 GiB
Save serializer split into smaller functions, debug information and safety checks retained 145 s 5.34 GiB
  • The serializer prototype reduced compile time about 40% and peak private memory about 49%. Its main serializer function shrank from 523 KB to 73 KB of WASM code.
  • LLVM control-flow restructuring fell from about 61 to 15 seconds; register stackification fell from 44 to 8 seconds.
  • Linking the same original compiled libraries with Emscripten -O1 took about 2 seconds versus approximately 205 seconds for the original build/link invocation at -O3. WASM size increased from 35,954,722 to 37,033,628 bytes (about 3%).
  • An unchanged compiled build took about 1 second; warm generation took about 4 seconds.
  • Other large functions include startup (335 KB), the entity-detail inspector (165 KB), and deserialization (133 KB).

These are exploratory local measurements, not Linux/container guarantees. Compiler runs used timing instrumentation; private memory was sampled. The serializer prototype compiled and linked but has not undergone save/load or gameplay validation. The faster link setting needs browser correctness and runtime-performance comparisons. Prototype files and measurements are retained locally under D:/prj/labelle-toolkit/wasm-investigation-evidence; attach reproducible scripts/results and the patch to the implementation PRs so validation does not depend on that machine.

Work packages

1. Versioned Docker builder image

  • Pin the base image and compatible Zig, Labelle CLI/assembler, Emscripten, Node and Python versions; record the toolchain inventory.
  • Install required system libraries, including libatomic1, and Chromium WASM smoke-test dependencies.
  • Run builds as a non-root user; include no credentials or game source.
  • Respect project.labelle/labelle.lock pins and document image/toolkit compatibility.
  • Keep toolchains in the image, persistent Zig local/global caches, Labelle package caches and Emscripten caches outside it, and game bundles/test reports as CI artifacts.
  • Document cache mounts, ownership, keys and invalidation by toolchain/platform/build configuration.
  • Publish versioned images to an agreed registry, pin consumers to immutable digests, and document upgrades and rollback.
  • Confirm/configure a Docker-compatible GitLab executor; shell executors do not apply the CI image setting. Document CPU/memory/concurrency limits using measured requirements.
  • Provide a GitLab CI example and adopt it in Flying Platform.
  • Keep Android SDK/NDK support in a separate future image/variant.

2. Reduce oversized engine/compiler functions

  • Implement and validate the serializer prototype in labelle-engine: move per-component entity collection and component writing into appropriately bounded functions so optimization does not expand everything into one oversized function.
  • Preserve serialization format, error propagation, allocation cleanup, component policies and ReleaseSafe checks.
  • Add targeted round-trip/regression coverage for representative FP components, marker components, prefab/built-in components and reference arrays; verify compatibility with existing saves.
  • Profile remaining large functions, including deserialization, generated startup and inspector code. Reduce excessive inlining/unrolled code where measurements justify it.
  • Update assembler templates/generation where needed; do not ship edits only to generated .labelle files.
  • Re-measure compile time, peak memory, output size and runtime behavior. Debug stripping alone is not the primary fix.

3. Faster CI/preview linking

  • Expose a documented WASM link-optimization setting independently of Zig's optimization/safety mode.
  • Evaluate Emscripten -O1 for CI/preview builds while retaining ReleaseSafe checks, assertions, pthread settings and required browser features.
  • Preserve an explicit optimized production/release path; do not silently change production defaults without runtime measurements.
  • Compare -O1 and -O3 using identical compiled inputs, measuring link time, output size, startup, representative gameplay/frame performance and memory.
  • Verify both threaded and single-threaded fallback bundles, browser smoke tests and assets/audio/settings/save-load behavior.

4. Independently compiled engine boundaries

  • Audit which dependencies are already separate cached libraries (including BGFX/C++) and which Zig engine code is imported as source into the game compilation.
  • Identify stable engine operations that can live behind explicit library/API boundaries, keeping FP-specific component/hook specialization in a smaller adapter.
  • Prototype a viable boundary and measure whether game-only edits can reuse the engine artifact. Moving source into another imported module alone is insufficient.
  • Validate target/ABI compatibility, debug-symbol behavior, threaded/fallback variants and cache invalidation.
  • Implement boundaries with demonstrated benefit, or document measured blockers/tradeoffs and an explicit disposition for this work package; do not silently omit it.

5. Releases, consumer pins and end-to-end validation

  • Release affected toolkit packages in dependency order and update Flying Platform's released dependency pins.
  • Adopt the versioned builder image and selected CI link profile in FP's GitLab job.
  • Publish reproducible cold, warm and changed-source benchmark results, separating image pull/startup, provisioning, generation, compilation, linking and smoke tests.
  • Verify a fresh container/runner needs no manually installed host toolchains, and subsequent fresh containers reuse persistent caches.
  • Compare peak compiler memory and safe job concurrency; containerization alone does not resolve oversized compiler workloads.

Acceptance criteria

  • Dockerfile, image build/publish automation, immutable image version and GitLab usage documentation are available.
  • Fresh-container builds and cache reuse work as a non-root user.
  • Serializer changes are implemented in toolkit source and pass save/load compatibility and regression tests.
  • Remaining oversized generated functions are profiled, with measured fixes and documented results.
  • Independent WASM link optimization is exposed and validated; CI/preview and production behavior are documented.
  • A separately compiled engine boundary is prototyped and evaluated, with implementation or explicit evidence-backed disposition.
  • Threaded/fallback bundles and Chromium smoke tests pass; runtime correctness/performance tradeoffs are recorded.
  • Affected toolkit releases are published and FP dependency/image pins are updated.
  • FP GitLab CI passes using the image; bundles, reports and cold/warm/changed-source measurements are retained.
  • Upgrade, rollback, cache invalidation and resource/concurrency guidance are documented.

Related prior work: labelle-cli#112 (Docker/Colima support), labelle-cli#206 and #210 (WASM scene materialization), and labelle-cli#331 (Android Docker limitations).

Updated by Codex, OpenAI LLM, at the owner's request to consolidate the image and all WASM build-performance improvements in this ticket.

Activity

  1. apotema
    changed the title [-]Publish a versioned Labelle WASM Docker builder image for GitLab CI[/-] [+]Speed up Labelle WASM builds: Docker CI image, compiler memory, linking and engine boundaries[/+] on Oct 30, 2026
  2. apotema
    added sub-issues on Oct 30, 2026

  3. apotema commented on Oct 30, 2026

    @apotema
    apotema
    on Oct 30, 2026
    ContributorAuthor

    Status update (2026-10-30)

    WP4 — independently compiled engine boundary: disposition: do not implement

    Prototype and measurements: labelle-engine#923 (write-up, single-threaded follow-up), draft PR labelle-engine#924.

    • C/C++ deps (bgfx/bx/bimg, imgui, stb, miniaudio) and emsdk libs are already separate cached artifacts.
    • All Zig (engine, plugins, packs, FP) compiles as one unit, and the engine is generic over game types (GameConfigWithYAxis(Renderer, …, Components, Hooks, Events)), so 94% of the 5.6 MB game-unit code is game-specialized. Only ~3% (~170 KB) is game-independent — the ceiling for any engine library boundary.
    • Prototype moved the largest such cluster (TMX/XML parser) behind a C ABI static lib: engine lib is reused on game edits, but the game compile stayed at 218–242 s vs 220.8 s baseline (noise), peak memory unchanged (~10.3 GiB), wasm +19.7 KB.
    • The real levers are WP2 (oversized generic functions) and WP3 (link level). Not validated: browser execution of the boundary path; Windows-only samples.

    Other work packages

    • WP1 — builder image: labelle-cli#529 → draft PR labelle-cli#530. amd64 + native arm64, non-root, apt pinned to a snapshot.ubuntu.com date, caches outside the image; PR CI green incl. end-to-end bundle + threaded/fallback Chromium smoke (amd64: 177 s cold, 23 s warm). FP GitHub Actions trial (moca-tecnologia/flying-platform-labelle#1095) is blocked on org Actions billing; local Docker run of FP pending.
    • WP2a — serializer split: labelle-engine#921. Engine change + golden/round-trip tests committed on perf/save-serializer-outline; full suite matches main (pre-existing Windows-only failures). A/B compile measurement in progress, PR to follow.
    • WP2b — remaining oversized functions: labelle-engine#922, not started (after WP2a).
    • WP3 — link optimization setting: labelle-bgfx#208 → draft PRs labelle-bgfx#209, labelle-web#28, labelle-assembler#821. LABELLE_WASM_LINK_OPT (unset = today's byte-identical command). FP, same compiled inputs: -O3 194.8 s / -O2 21.7 s / -O1 7.8 s link; wasm +3% at -O1 (+1% gzipped). Threaded and fallback Chromium smokes pass at -O1 and -O3. Gameplay/frame-time, save/load and audio not yet validated. Found: labelle-web's best-effort wasm-opt -O3 currently fails validation and is skipped in production exports (separate issue to file).
    • WP5 — FP adoption: moca-tecnologia/flying-platform-labelle#1094, after releases.

    https://claude.ai/code/session_01Hpt1haxcboZPznpG1KqDRS

  4. apotema commented on Oct 30, 2026

    @apotema
    apotema
    on Oct 30, 2026
    ContributorAuthor

    WP1 update: the WASM builder image moved out of labelle-cli into Flying Platform as a private image (moca-tecnologia/flying-platform-labelle#1101). #530 and #529 are closed as moved.

    Toolkit-relevant findings from the image runs:

    • FP's cold WASM compile peaks at 11.5 GiB in a Linux container (one changed script: 10.8 GiB). That is still the engine#921 problem.
    • The generated build.zig.zon pulls emsdk 4.0.9 as a Zig package and installs it inside each checkout (.labelle/bgfx_wasm/zig-pkg/, about 1.4 GB) instead of using the provider's EMSDK. It is a toolkit issue and not filed yet.
    • Under Docker Desktop, a Windows host bind mount breaks Zig's cache (AccessDenied on rename).

    https://claude.ai/code/session_01Hpt1haxcboZPznpG1KqDRS

  5. apotema commented on Oct 30, 2026

    @apotema
    apotema
    on Oct 30, 2026
    ContributorAuthor

    Toolkit bugs found during the GitLab CI trial (moca-tecnologia/flying-platform-labelle#1102) and the builder-image runs:

    GitLab trial result: FP bundle at O1 with engine #926+#925 (on v4.2.0-rc.2), bgfx #209, web #28 and assembler #822: 251 s, 4.26 GB peak compiler memory (was 10.4 GiB), threaded and fallback smokes pass.

    https://claude.ai/code/session_01Hpt1haxcboZPznpG1KqDRS

  6. apotema
    added 8 commits that reference this issue on Oct 31, 2026

    feat(wasm): link optimization setting independent of -Doptimize (#208) (

    d211fd1

    feat(export): wasm-opt level follows LABELLE_WASM_LINK_OPT (labelle-b…

    c7321cc

    feat(wasm): LABELLE_WASM_LINK_OPT in the in-tree emcc hooks (labelle-…

    5e6bd51

    chore: release labelle-bgfx 0.38.0 for WASM link optimization setting (…

    d00f8eb

    chore: release labelle-web 0.5.0 for WASM link optimization (#30)

    e5dcac4

    chore: release labelle-assembler 0.125.0 for WASM link optimization (#…

    f7db04e

    release: labelle-imgui v0.14.8 — bridges honour EMSDK with EM_CACHE /…

    32657d6

    chore: release labelle-assembler 0.126.0 for lazy root build deps and…

    5718043

  7. apotema commented on Oct 31, 2026

    @apotema
    apotema
    on Oct 31, 2026
    ContributorAuthor

    Overnight release train — done (2026-10-31)

    Everything below is released or merged, with CI green on the exact commits that were tagged/merged.

    Toolkit releases

    Package Version What
    labelle-audio v0.5.1 emsdk resolved from a valid EMSDK; headers from $EM_CACHE with sysroot_install.stamp readiness (#12)
    labelle-bgfx v0.38.1 no secondary 1.4 GB emsdk install with a valid EMSDK; stamp-based sysroot readiness; needs audio 0.5.1 (#210)
    labelle-imgui v0.14.8 same emsdk rule in both bridges; lazy root sokol dependency (#49)
    labelle-assembler v0.126.0 lazy root build deps + lazy-safe emsdk preflight (#824); inspector outlining (#822)
    labelle-engine v4.2.0-rc.3 (prerelease) serializer split (#926) + loader/scene-registration outlining (#925) on the 4.2 line
    labelle-cli v4.1.3 concurrency-safe downloads into the shared labelle home (#533)

    Earlier the same day: labelle-bgfx v0.38.0, labelle-web v0.5.0, labelle-assembler v0.125.0 (LABELLE_WASM_LINK_OPT), labelle-registry web 0.5.0 entry.

    Flying Platform (moca-tecnologia/flying-platform-labelle)

    • #1105 bgfx 0.38.0 + web 0.5.0, CI WASM jobs link at O1 (production stays -O3)
    • #1106 / #1108 / #1111 builder image pins: 4.1.2-2 → 4.1.2-3 → 4.1.3-1 sha256:4e20e902df8b9c2f50cdfeb4ab919fd1ede385ec53d6682f85671752620edf62
    • #1107 image 4.1.2-3: astcenc 5.5.0 seeded into the cache (sha256:e4277f4d65ab402dc3f508e5c15c3f32ac8bdadf39f8058079f2fec3239fdd3a)
    • #1109 engine 4.2.0-rc.3, bgfx 0.38.1, assembler 0.126.0, imgui 0.14.8; debug plugin re-vendored
    • #1110 CLI 4.1.3; image 4.1.3-1 (CLI 4.1.3 + assembler 0.126.0 seeded)
    • Every image was published through the promote gate (sanity and test markers on the same digest).

    Measured effect (FP WASM, ReleaseSafe)

    Workarounds log

    1. FP#1109's first GitLab run failed: two parallel jobs raced downloading assembler 0.126.0 into the shared cache (after fp-ubuntu went to concurrent = 3). Retried with an empty commit (passed), then fixed at the source in labelle-cli v4.1.3 (fix(cache): concurrency-safe downloads into the shared labelle home #533).
    2. labelle-engine#927 (rc.3 release PR into the 4.2 branch) was merged by fast-forwarding the branch to the tagged, CI-tested commit instead of squash-merging, so the branch points at exactly what was tagged.
    3. labelle-cli v4.1.3 was tagged before the release/4.1.3 E2E run finished; that run later passed on all platforms.
    4. The post-merge main E2E for the 4.1.3 version bump failed once on Windows in provider_watch_e2e.py ("the refused session rewrote the status file"); the same test passed on the fix commit and the release branch. Re-ran the failed job: green.
    5. FP's GitHub build-and-deploy check fails because GitHub Actions billing for the org is failing; it is not a required check and nothing was bypassed. Needs billing fixed.
    6. While stopping the emsdk 6 evaluation agent, the main session deleted .heavy-lock assuming it was that agent's; the image agent had just taken it. Its run continued unaffected.
    7. Left on disk: epic-528/fp-pin-4123 and epic-528/fp-pin-4131 worktrees (in use as session working directories; safe to delete), epic-528/emsdk-6.0.13 + emsdk6-work, and the 30 GB epic-528/docker-cache-823/cache.ext4.

    No admin overrides, protection bypasses, force-pushes to others' branches, disabled tests, or republished versions.

    Still open

    • Evaluate upgrading Emscripten 4.0.9 → 6.0.13 across the toolkit (partial results) #532 — evaluate Emscripten 6.0.13 (partial results; no CI gain, ~20–25 % faster -O3 links).
    • fp-docker-pr runner (PR coverage in the image) and moving fp-docker off the ci user — box work, see FP docs/ci-docker-runners-nixos.md.
    • In-browser gameplay check of save/load, inspector, audio and settings on the new builds.
    • labelle-web#29 (wasm-opt validation at -O2/-O3), labelle-cli#531 (SHA engine pin crash).

    https://claude.ai/code/session_01Hpt1haxcboZPznpG1KqDRS

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions