Skip to content

Deploying to open-compute 0.4.0 fails: 14.8 MB single-bundle Worker exceeds the management API's 8 MiB per-part upload limit #24

Closed

Description

@Azathothas

Diffinery: v0.2.0, commit 059a44f (main, Oct 2026), Apache-2.0
Target platform: open-compute 0.4.0 (release=0.4.0, git_revision=213219468bd64d3c47138333817ead2f458c8eb0, workerd 2026-09-30), Debian 13, Linux 6.12.113+deb13-amd64, x86_64
Command: ocd cf --target <instance> -- deploy --mode production (cf 1.0.0-beta.12, per open-compute's documented deploy path)

Summary

Diffinery declares celld as its non-Cloudflare target and is advertised as "one Worker, everything on Cloudflare". open-compute is the other self-hostable Workers-compatible platform, and its product surface (Workers, DO+SQLite, D1, R2, Queues, Workflows, Containers, WebSocket hibernation, cron) covers Diffinery's binding list except the documented-optional ones (ratelimits, send_email).

The deploy fails at transport, not compatibility: Diffinery's Worker bundles to a 14.8 MB single ESM file, and open-compute 0.4.0's management API rejects any multipart part over 8 MiB (413, code 10027 "Request body is too large"). cf deploy (which delegates to wrangler) sends the bundle as one part, so the upload cannot succeed. The failure is in open-compute's ingress, and I filed the mechanism there separately; see: elliothux/open-compute#195
This issue tracks what Diffinery can do about it, because at least two of the three fixes are on Diffinery's side.

Reproduction

1. Build the bundle

git clone https://github.com/codician-team/diffinery.git && cd diffinery
pnpm install && pnpm -r build

cd apps/diffinery
npx esbuild src/index.ts --bundle --format=esm --platform=neutral \
  --outfile=/tmp/diffinery-bundle/worker.js \
  --external:cloudflare:* --external:node:* --external:@peculiar/* \
  --loader:.wasm=file --loader:.bin=file

Result: worker.js = 14,775,975 bytes (14.8 MB) plus five emitted side files (.wasm, .br.bin) totalling ~530 KB. The bundle imports only cloudflare:workers, cloudflare:workflows, cloudflare:sockets, and node:* builtins (node:crypto, node:fs, node:http, ...), all of which open-compute's workerd provides under nodejs_compat.

2. Deploy and fail

ocd cf --target local -- deploy --mode production
# ├  Deploy
# │  Total Upload: 14.75 MiB / gzip: ~4 MiB
# ┌ APIError
# │ [10027] Request body is too large
# │ 413
# └

A minimal raw upload reproduces it without cf:

curl -X PUT "$BASE/client/v4/accounts/$ACCT/workers/scripts/diffinery" \
  -H "Authorization: Bearer $TOKEN" \
  -F 'metadata={"main_module":"worker.js","compatibility_date":"2026-10-01","compatibility_flags":["nodejs_compat"]};type=application/json' \
  -F 'worker.js=@worker.js;type=application/javascript+module'
# -> 413 {"code":10027,"message":"Request body is too large"}

The boundary is ~8 MiB per part (measured cutoffs within a few hundred bytes of 8,388,608 across three bisect runs): single modules just under the cap upload, just over return 413. A 16.7 MB body with two ~8 MiB modules is accepted, so the platform demonstrably supports large multi-part bundles; only the single-part path is capped. Full measurement transcripts and repro script in the open-compute issue (companion report).

3. What does work on open-compute (so the gap is only this upload path)

Verified independently on 0.4.0 before this failure:

  • PUT .../workers/scripts/<name> with small multi-module bundles: accepted, and the worker runs at http://<script>.<instance>.localhost:8787/ (endpoint discoverable via GET /client/v4/accounts/{a}/open-compute/workers/{s}/endpoints).
  • KV smoke worker deployed and exercised via ocd cf deploy (cf 1.0.0-beta.12).
  • ocd capabilities --json: durable_objects supported_with_deviation, d1, r2, queues, version_metadata, websocket_hibernation, containers supported; workflows blocked-for-target with deviation OC-WORKFLOW-001 but documented as executing on a local SQLite authority; rate_limiting unsupported (Diffinery already handles this via RATE_LIMITS=durable, the same fallback it ships for celld); send_email absent (Diffinery degrades to logged-skipped email, same as celld).

Why Diffinery should care

Diffinery's positioning is "deploy it to your own account" on Cloudflare, with celld as the leave-Cloudflare path. open-compute is a second self-hosted target with an API-compatible management surface (/client/v4), a single binary, no bucket requirement, and 2.9k stars. Today Diffinery cannot deploy to it for a transport reason, not an API or runtime reason, and most of the plausible fixes are Diffinery-side.

Suggested directions

  1. Split the bundle. open-compute accepts multi-part uploads with a real import chain (verified: a 3-module, 10 MB deploy with import statements deployed and executed). esbuild --splitting with ESM chunks under 8 MiB, or a small post-build splitter that cuts the bundle at module boundaries, would clear the cap with no runtime change. The 14.8 MB bundle splits trivially into 2 chunks.
  2. Ship an assets-style path for the big blobs. The .wasm/.br.bin side files (~530 KB) are fine as parts, but if future bundles grow, moving static data out of the script (R2/static assets) keeps the JS part small.
  3. Add an open-compute target to scripts/ (a celld-config.mjs equivalent that emits the multi-part upload) once the cap is documented or fixed upstream, mirroring the existing celld integration and its CELLD_ABSENT_BINDINGS-style unit test. The two optional bindings (ratelimits → RATE_LIMITS=durable, send_email → skip) are already optional in Env, which the celld work proved.
  4. File/track the upstream cap (companion issue against elliothux/open-compute): make the ingress limit configurable or derive it from workers.max_bundle_bytes (34 MiB).

Environment

  • Host: Debian 13 container, 8 cores, 14 GB RAM, Node 22.23.4, pnpm 10.34.9
  • Diffinery: git+https://github.com/codician-team/diffinery@059a44f, pnpm -r build clean, typecheck clean, full unit suite green (1249 app + 110 runner + 197 workflow-yaml + 51 git-core tests) before the deploy attempt
  • open-compute: ocd 0.4.0 user scope, 127.0.0.1:8787, reproduced on both the default and a virgin instance
  • cf: 1.0.0-beta.12 (project-local, per open-compute's ocd cf requirement); wrangler (bundled with cf) performs the build step

Activity

  1. codician-dev commented on Nov 4, 2026

    @codician-dev
    Contributor

    Thanks for the thorough report, and for bisecting the 8 MiB boundary. That made this quick to fix.

    #25 (merged) moves the two bulky generated pieces out of the Worker's main module into text modules, which upload as separate parts:

    • The docs content (pages, images and the CLI, editor and plugin downloads, about 4.4 MB) is now one JSON text module, parsed once at startup.
    • Shiki's grammars (about 3 MB, which the bundler had inlined) are now one text module per grammar, parsed on first use.

    With wrangler deploy --dry-run after pnpm -r build, the main module goes from 14.4 MB to 6.8 MB, and no other module is over about 4.5 MB, so every part fits under open-compute's 8 MiB cap. The total upload is unchanged.

    One setting on your side: the source map is still a single ~12 MB part, so set "upload_source_maps": false in wrangler.jsonc for open-compute. The self-hosting docs (Architecture and Configuration) now say so.

    We haven't run a full deploy on open-compute ourselves, so if anything else trips (Workflows under OC-WORKFLOW-001, for example), tell us here. I'll leave this open until you've had a chance to try it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions