Skip to content

Wasm.Browser: policy 250 ms deadline includes Worker cold-start; Firefox refuses DeadlineExpired on hosted 4-vCPU #727

Open

Description

@EHotwagner

Observed: hosted run FS-GG/FS.GG.Templates 39874053570 (installed-C, ubuntu-24.04 hosted, 4 vCPU) ran the Workspace.Template 0.19.2 receiver on FS.GG.Wasm.Contracts 0.3.0 and FS.GG.Wasm.Browser 0.3.0.

  • In Firefox, the first case was refused with Refused: DeadlineExpired right after "Load policy".
  • Chromium passed 9/9. WebKit never ran, because the run stops at the first failure.

Cause (from the trace):

  • The policy's 250 ms deadline starts when the request is queued.
  • Wasm.Browser creates a fresh Worker at that point, and the Worker then loads about 38 separate ES modules.
  • Fetching those modules alone took about 252 ms, so the budget ran out before compilation started.
  • The failure depends on CPU speed: it passes locally on 24 CPUs, and Chromium passed every cold load. It is one sample. Report: Qualify the first clean downloaded shared-game-reference journey .github#4710 (comment)

Why it matters: this is a product defect, not a test artifact. A Firefox user on a modest machine gets a refused policy on first load. And because the producer's browser tests run Chromium only, nothing upstream covers Firefox or WebKit.

fdev decision on the fix direction (the owner designs the details):

  1. The policy deadline bounds policy work only, not Worker infrastructure startup. Start the 250 ms clock when the Worker reports ready, and give Worker startup its own separate, bounded readiness timeout. Refusal on that timeout must be distinguishable, e.g. WorkerStartTimeout, not DeadlineExpired.
    • Update the lifecycle model eng/wasm-shared/lifecycle.qnt and its checks so they stay in agreement.
    • Optionally, reduce startup cost by bundling the Worker into a single module, if that's cheap.
  2. Add Firefox and WebKit to the producer's browser tests. Include a cold-load case under a hosted-like CPU limit (e.g. 4 CPUs, or Playwright CPU throttling) that records the startup time per browser, so there is visible margin under both bounds.
  3. Publish new FS.GG.Wasm.* versions under the normal release contract. Then the Templates receiver re-pins them and a Workspace.Template patch release follows; those are tracked from Qualify the first clean downloaded shared-game-reference journey .github#4710.

Blocks FS-GG/.github#4710 (installed-C 27/27). Filed by fdev.

Activity

  1. EHotwagner commented on Oct 23, 2026

    @EHotwagner
    ContributorAuthor

    fdev 2026-10-23: Source slice merged; publication admission requested. Nothing has been published or tagged.

    Accepted source slice. PR #728 was merged as 15e287d7c8884f088b7db41140f108f3f0136043 (tree 48d23dd8bc8813877ddbbbf039b87087843f3ca6) at reviewed head 5cbcbe7. An independent fsgg-critic-best review accepted that exact head (decision), after a REJECT on 9f202d9 that is now resolved (CI headroom). All 34 PR checks passed.

    What changed

    • Policy deadline vs. startup: the 250 ms profile deadline now covers guest policy work only.
      • The load command is still posted when the Worker is created.
      • The Worker's first Compile receipt for that command counts as its readiness report.
      • Until that report arrives, the load is in phase start under its own startup bound. The default is 5,000 ms; HostSettings.withWorkerStartTimeout selects 1–60,000 ms. A caller's enclosing deadline still caps it.
      • If startup expires, the load is refused with the new DeliveryReason.WorkerStartTimeout (added as the last case): Discarded, TimedOut/Compile, not dispatched. The Worker is terminated and gains no game authority.
      • Policy expiry after readiness is still DeadlineExpired.
    • Why 5,000 ms: it is 20 policy budgets. It is at least 50× the hosted cold-start maximum measured below, and a Worker that never starts is still refused within seconds.
    • Model: lifecycle.qnt and the qualification models were updated to match:
      • a start phase and new invariants;
      • 5 new Quint tests and 5 new fixed scenarios;
      • regenerated traces (58 default, 16 selected), with .NET/Fable correspondence and three new mutation checks.
    • Pre-existing Firefox defect found and fixed:
      • Firefox can run a setTimeout callback before the host clock passes its due time (36 of 300 probes; 0 of 300 in Chromium and WebKit).
      • When that happened, the expiry was never re-armed, so a guest stuck in a loop could hang without being retired.
      • The connected interpreter now re-arms the remaining time instead.
    • Browsers: the package-consumer suite (38 cases) and the raw conformance suite (6 cases) now run in Chromium, Firefox and WebKit.
    • Worker bundling: a single bundled Worker has shipped since 0.5. The 0.3 receiver that failed had the unbundled ~38-module Worker, so re-pinning picks up the bundle.

    Hosted evidence (ubuntu-latest, 4 vCPU; 20 fresh-profile cold loads per browser; times in ms; startup = Worker creation → readiness report; policy load = readiness → loaded)

    Run Browser Startup p50 / p95 / max Policy load p50 / p95 / max
    main push 39891458102 at 15e287d Chromium 153 22.2 / 23.0 / 27.4 4.0 / 4.6 / 5.3
    Firefox 155 46 / 57 / 99 4 / 5 / 6
    WebKit 26.6 39 / 44 / 44 6 / 7 / 7
    PR head 39888632477 at 5cbcbe7 Chromium 24.8 / 27.5 / 35.8 4.4 / 4.8 / 5.7
    Firefox 50 / 76 / 100 4 / 5 / 5
    WebKit 41 / 48 / 61 7 / 7 / 8
    • Margin: the worst startup was 100 ms against 5,000 ms. The worst policy load was 9 ms (in an earlier run) against 250 ms. Even the whole load end to end, at most 104 ms, would now fit inside 250 ms.
    • Main push run 39891458102: succeeded on attempt 1 in 22m33s. The cold-load artifact is wasm-cold-load-39891458102-1 (id 12475009173, sha256:9d64176a…f9f471).
    • Local 4-CPU Ubuntu 24.04 container: the same suites passed 114/114 and 18/18.

    Publication admission request. Please admit one normal coherent publication through release-wasm.yml, from the protected eligible custody of main push run 39891458102 attempt 1.

    Field Value
    Candidate (producer and executor) 15e287d7c8884f088b7db41140f108f3f0136043, tree 48d23dd8bc8813877ddbbbf039b87087843f3ca6
    Eligible artifact wasm-eligible-39891458102-1-0.7.0, id 12476030732, sha256:e69cdf37b1af06197469cece0a8abf3c15e6a50a683672fbcf52b5f513ff31b5
    FS.GG.Wasm.Contracts 0.7.0 6a77863392f371f07e5b5fd7af94bdf45c7c87dea976bbc3b20336b59df70618
    FS.GG.Wasm.Browser 0.7.0 1d5e9df281357919a1749f0bf28d1786d36192cb768c756d4e4bec3fc7aae89e
    fsgg-wasm-sdk-0.7.0.tar.gz 090658bd29c79500ed6f3f3defe89a14f7b8330b39586cd01880429258a99cb2
    release-manifest.json 56e585dbffc2941a504aec75c7fe99c63fa166d03a65eb650a65ba1550a53407
    Tag wasm/v0.7.0
    Feeds org and public, under the normal release contract

    Why a minor version (0.7.0): both packages are ApiCompat-compatible with published 0.6.0 (served-archive baseline, no suppressions). Still, Browser gains a public reason case and HostSettings members, and load timing behaviour changes, so this is a minor version, not a patch.

    Held remainder (#727 stays open)

    1. Publication of the set above. It needs your admission, and no effect has been taken.
    2. Installed acceptance of 0.7.0.
    3. The Templates receiver re-pins to 0.7.0, followed by a Workspace.Template patch, both tracked from Qualify the first clean downloaded shared-game-reference journey .github#4710. The receiver should keep loading without an enclosing deadline, because an enclosing deadline also caps startup.
    4. Installed-C run 3 under a new admission.

    Residual notes (not blocking)

    • Startup expiry under Sc2SupervisorV1 is reached only through the shared F# expiry path. No selected model sample or fixed scenario covers it.
    • If a starting Worker sends a protocol-violating invocation receipt, the load is settled first and the Worker retired after. This behaviour predates Bound WASM Worker startup separately from the policy deadline (#727) #728.
    • The browser install step is now bounded at 20 minutes with one retry. A very slow apt mirror could still fail it, though it now fails clearly at that step instead of the job timing out.

    Knowledge: captured in Atlas as game/wasm-worker-startup-bound, digest b091d938…, in docs/atlas.

    fdev

  2. EHotwagner commented on Oct 23, 2026

    @EHotwagner
    ContributorAuthor

    fdev 2026-10-23: ADMITTED: publication of FS.GG.Wasm 0.7.0 (Contracts 6a778633…, Browser 1d5e9df2…, SDK 090658bd…, manifest 56e585db…) from eligible artifact 12476030732 (e69cdf37…), binding 941c5184…, executor 15e287d.

    The independent GO review gave GO-WITH-CONDITIONS. It checked the release contract, the hashes, the absence of 0.7.0, ApiCompat, the fix against the decided direction, the Firefox re-arm, and the cold-load evidence (Chromium, Firefox and WebKit worst startup at 27, 99 and 44 ms).

    Conditions, applied:

    1. No tag push. fdev dispatches release-wasm.yml mode=promote exactly once, with the reviewed tuple. The workflow creates the tag itself.
    2. Merges to Game main are held until the run completes. Every job re-checks that main is 15e287d.
    3. On failure, no re-run. Recovery is a mode=recovery dispatch, after nuget.org indexing if any push may have landed.
    4. This issue stays open for installed acceptance and the Templates re-pin.
  3. EHotwagner commented on Oct 23, 2026

    @EHotwagner
    ContributorAuthor

    fdev 2026-10-23: FS.GG.Wasm 0.7.0 is PUBLISHED. release-wasm.yml run 39894673177 succeeded on attempt 1: preflight, admission, begin, org, public, assets and complete.

    fdev readback:

    • nuget.org flatcontainer lists 0.7.0 for both FS.GG.Wasm.Contracts and FS.GG.Wasm.Browser;
    • the tag wasm/v0.7.0 → 15e287d;
    • the release is non-draft, with 4 assets.

    The merge hold on Game main is lifted.

    Next (#4710 path): re-pin the Templates receiver (0.3.0 → 0.7.0). That means:

    • regenerate the lock file from nuget.org;
    • update the SDK hash in materialize-assets.py to 090658bd… and rebuild the guests;
    • adjust the module-graph walk for the bundled Worker;
    • keep no enclosing deadline on the load.

    Then a Workspace.Template patch release and installed-C run 3. This issue stays open until installed acceptance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions