Skip to content

test(aaas): multi-process fault-injection drill and the #275 closure (A1.4) - #681

Open
Danceiny wants to merge 4 commits into
feat/aaas-a1-tenant-routerfrom
feat/aaas-a1-fault-injection
Open

Danceiny wants to merge 4 commits into
feat/aaas-a1-tenant-routerfrom
feat/aaas-a1-fault-injection

Conversation

@Danceiny

Copy link
Copy Markdown
Owner

Refs #674

A1.4 closes A1 of the AaaS charter. The exit evidence the RFC asks for is "offline concurrency and fault-injection suites on isolated state roots" and "every #275 checklist bullet mapped to a passing test or a recorded decision". This PR adds the first as a drill with real processes and records the second in the RFC. It lives outside the kernel and is not wired into a runtime path. state-ledger.ts and the kernel manifest are untouched, and no dependency is added.

Stacked on #680 (the base of this PR is its branch, so the diff shows only A1.4). When the stack below it merges I rebase onto main and retarget; there is nothing to decide in between.

The drill (tenant-state-drill-tests.ts, run-all §92, about 35 seconds)

Four replica processes, each on its own state root, contend for two tenants through one lease file and one durable directory. They write numbered events, snapshot every 120 ms, and log what they did to files that survive their own death. The drill then:

  • kills the owner of tenant a with SIGKILL, twice;
  • freezes the owner of tenant b with SIGSTOP past its lease, waits for a takeover and a pin, and resumes it;
  • hands tenant a off with SIGTERM (the graceful path).

A verifier then takes each tenant over and compares the ledger with the logs:

  1. I1 The lease epochs are consecutive: none granted twice, none skipped.
  2. I2 Each epoch has at most one writer.
  3. I3 No write is acknowledged after the next owner took the lease. On a real clock, the local gate does its job for the resumed process.
  4. I4 The restored ledger is intact, holds only this tenant, and is exactly the lineage the durable copies describe: every pinned snapshot holds exactly the writes acknowledged before it began, and nothing else is there.
  5. I5 A graceful handoff loses nothing; a crash loses only the writes after the last snapshot (33 acknowledged writes across the two tenants in the recorded run).
  6. I6 The two tenants never see each other's writes.

It is a simulated drill: it is engineering evidence for A1 and counts for nothing in A3 or A4.

Also in this PR

  • The data(state): track D-15 multi-user replication and claim fencing triggers #275 closure (RFC §7): each bullet now names the passing suites, and the one thing not built is a recorded decision. Choosing an earlier snapshot on purpose is an operator action and waits for A3; the restore path already falls back to an older verified snapshot.
  • A window D-a cannot close, pinned. tenant-state-copies-tests X3 reproduces it: a frozen-clock owner lands a snapshot between its successor taking the lease and pinning its base, and that snapshot is restored. The test carries a note on how to flip it once the window is closed by D-b or by a seal on the previous epoch. RFC §4.3 states the limit.
  • Three small fixes found in review, each with a test and a mutation check. Opening a tenant that is being handed off now says it is draining, instead of reporting it held by this very replica. A consumer's ledger.close() on the guarded view no longer closes the ledger the router owns. Durable keys pad the epoch to eight digits but the restore regex matched exactly eight, so a loop that burned a hundred million epochs would have hidden every copy; it now matches eight or more.
  • RFC §4.2 item 3 now states the restore rule the code implements (the newest verified snapshot of the newest lineage, pinned by the new owner's base) instead of "the prefix of the highest epoch the lease store records".
  • The D-15 row of architecture §10 now says A1 proved the semantics offline and what remains (A3, A4).

Evidence on 3c08ed670d9139b0e7a8e32195d39b58bd78116e (local Node v26.9.0)

  • cd ts && npx tsx scripts/tenant-state-drill-tests.ts: exit 0, 2,766 assertions in the recorded run (2 SIGKILLs, 1 SIGSTOP past the lease, 1 graceful handoff, 2,734 acknowledged writes).
  • Mutation check: I broke five guards in the real sources, one at a time, and the drill went red every time. They were the local fence, the lease store's compare-and-set, the epoch moving forward, the newest-first lineage, and the final flush of a graceful handoff. The router fixes and the earlier slices carry their own mutation checks.
  • tsc --noEmit, smoke.ts, g5-guard.ts, kernel-manifest-gate.ts: exit 0, with a zero-diff kernel manifest. All four tenant-state suites pass (28, 26, 21 checks and the drill).
  • Full scripts/run-all-tests.sh: green on the A1.3 commit underneath (ALL SUITES GREEN, exit 0). On this head it was still running locally when I opened the PR, so CI on Node 22 and 24 is the first full result for it; this slice adds tests, docs and three small router and restore fixes.
  • docs i18n and readability, run-all wiring test: OK.

What is still open after A1

  • A2 and A3 are separate phases: seam-derived tenants and quotas, then the intranet tier, the boundary ADR and the MySQL and S3 bindings. The facts from hotel-be that shape them are in a local branch I have not pushed, because they describe HotelByte infrastructure and this repository is public.
  • Ledger file only. Files beside the ledger are not replicated yet (RFC gap 6, fact F-2).
  • Roadmap row. It still reads "A1 under way" until docs(aaas): record A0 RFC acceptance — ADR-15 restated as keyed single writer #677 merges; I will update it then, so the two do not edit the same sentence.

Review ask. Is A1 complete for you? If yes, the next step is A2.

huangzhen.0505 added 4 commits October 28, 2026 11:45
…(A1.4)

A1 exits only with evidence from real processes under kills, a pause past the lease and a graceful handoff, so the drill compares each tenant's final ledger with what the replicas logged, and the RFC maps every #275 bullet to a passing suite. It also pins the one window mode D-a cannot close. Outside the kernel; engineering evidence only.

Refs #674
…er cannot close the router's ledger

While a handoff drains, this replica still holds the lease, so opening the tenant would only report it held by itself; it now says it is draining. The router owns the ledger's lifecycle, so close on the guarded view does nothing.

Refs #674
The RFC said restore reads the prefix of the highest epoch the lease store records; what A1.2 builds is the newest verified snapshot of the newest lineage, pinned by the new owner's base, so the two now say the same thing.

Refs #674
Keys pad the epoch to eight digits but the restore regex matched exactly eight, so a loop that burned a hundred million epochs would have hidden every copy; it now matches eight or more.

Refs #674
@Danceiny

Copy link
Copy Markdown
Owner Author

Local full regression on 3c08ed670d9139b0e7a8e32195d39b58bd78116e (Node v26.9.0): ALL SUITES GREEN, exit 0, including run-all §89–§92: the three suites pass (28, 27 and 21 checks) and the drill reports 2,306 assertions (2 SIGKILLs, 1 SIGSTOP past the lease, 1 graceful handoff, 2,274 acknowledged writes, 29 lost in crashed tails). The PR body says this run was still going when the PR was opened; it has finished.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant