Skip to content

pool: bring a claim's volumes up concurrently - #77

Merged
CMGS merged 2 commits into
mainfrom
feat/parallel-volume-attach
Aug 13, 2026
Merged

pool: bring a claim's volumes up concurrently#77
CMGS merged 2 commits into
mainfrom
feat/parallel-volume-attach

Conversation

@CMGS

@CMGS CMGS commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Multi-volume claims paid a strictly sequential attach+settle+mount per
volume (~60-90ms marginal each, measured on bare metal in the #72 round).
The per-volume pipeline — write-ahead marker (rw), disk attach, sysfs
settle, guest mount — now runs concurrently across a claim's volumes via
errgroup; cocoon serializes the hypervisor attach per VM internally, so
what overlaps is the CLI subprocess spawns, the 5-10ms device settle
waits, and the guest mount execs.

  • Volume-less and single-volume claims are untouched (0 → early return,
    1 → inline, no goroutine).
  • The response echo and persisted entries keep request order; per-volume
    ordering (marker strictly before attach before mount) is unchanged.
  • Failure keeps fail-fast semantics: first error cancels the group, the
    claim fails, the VM is destroyed, holds release through the existing
    defer. One deliberate delta, pinned by test: a failed multi-rw claim
    now leaves every rw image marked (all markers are written up front),
    not just those before the failure point — the conservative direction
    of the block-don't-heal contract.
  • The concurrency pin has teeth: the new mixed 4-volume test gates every
    attach on a rendezvous that deadlocks under sequential apply (verified
    by temporarily reverting to a loop — the test times out — then
    restoring).

Gates: build, -race across the module (20x on the apply tests, no
flakes), dual-GOOS golangci-lint 0 issues, gofmt clean, asl dual-GOOS
zero findings. Bare-metal A/B for the multi-volume latency claim follows
as a comment.

Hardware verdict (see the A/B comment for full data): the win is gated on
guest vCPU count — the overlapped work is guest-CPU-bound. 1.00x on the
1-vCPU small tier, 1.33x at 2 vCPU, 1.70x (2.55x marginal per-volume) at
4 vCPU. Single-volume and release paths unchanged on every tier. The round
also surfaced pre-existing #78 (sysfs-vs-devtmpfs bring-up race), fixed
separately.

Cocoon serializes the attach itself per VM, so the win is overlapping
the CLI spawns, device settle waits, and guest mounts across volumes.
Single-volume claims run inline; per-volume ordering (marker before
attach before mount) is unchanged and pinned.
@CMGS

CMGS commented Aug 13, 2026

Copy link
Copy Markdown
Contributor Author

Hardware A/B on bare metal — the win is gated on guest vCPU count

Bare metal (16 core), cocoon master-07f1d69, CH v54, template rt:24.04,
none-lane pool warm=4, four 64MiB ext4 read-only catalog entries. Every sample
is settled (drain to claimed==0, refill to target, 2s idle) and warm-tier
proven per sample via the claims_warm counter delta; the client connects once
per arm and the first cycle is discarded.

Four-volume claim, default mounts, all four mounts asserted per sample:

tier (guest vCPU) main ce55029 p50 branch p50 Δ p50 speedup same-binary drift (mean)
small (1) 118.92 ms 120.37 ms +1.45 1.00x A/A 3.45
medium (2) 104.70 ms 78.74 ms −25.96 1.33x A/A 0.91, B/B 0.06
large (4) 121.92 ms 71.66 ms −50.26 1.70x A/A 2.34, B/B 0.64

Marginal cost of volumes 2–4, (p50(4vol) − p50(1vol))/3:

tier main branch
small 25.96 ms/vol 26.36 ms/vol unchanged
large 27.70 ms/vol 10.87 ms/vol 2.55x

Single-volume control (the inline case 1 path) — no regression:

tier main p50 branch p50 |A−B| p50 drift
small (16/arm) 41.04 ms 41.30 ms 0.26 A/A 0.77, B/B 1.29
large (40/arm) 38.82 ms 39.06 ms 0.24 run-mean spread 1.83 / 3.57

Release is unchanged on every tier (|A−B| mean 0.72 / 1.32 / 0.91 ms) — as
expected, teardown is untouched for read-only volumes.

Why the default tier shows nothing

An instrumented build (diagnostic only) logged per-volume offsets: on main the
serial pipeline is attach 5–7 ms + guest mount 19–36 ms per volume, four
back to back → ~119 ms. On the branch all four goroutines enter within 0.04 ms
and the mounts genuinely overlap — but on a 1-vCPU guest each mount stretches
from ~25 ms to 79–119 ms, so the wall is unchanged. The overlapped work is
guest-CPU-bound: the speedup tracks the guest vCPU count (1.00x at 1, 1.33x at
2, 1.70x at 4; 2.55x on the marginal per-volume cost). The four attaches still
complete 1–2 ms apart, so something downstream serializes part of the attach —
at 6–13 ms it is not the cost that matters here.

Pre-existing bring-up race, surfaced by this round

mount: /volumes/vol-X: /dev/vdY is not a valid block device → claim fails 500,
on 5/141 four-volume attempts (3.5%; main 2/58, branch 3/83 —
indistinguishable), 0/112 single-volume. Reproduces on ce55029, so this PR
does not introduce it — but multi-volume claims will hit it more once they are
worth using. findVolumeDevice matches /sys/block/<dev>/serial and returns
/dev/<dev>, while sysfs publishes the disk before devtmpfs creates the node.
Filed separately with a fix direction. Every failed claim settled cleanly
(claimed==0, warm back at target), and all four read-only fixture sha256s
were unchanged from creation through every phase.

@CMGS
CMGS merged commit d60f51d into main Aug 13, 2026
1 check passed
@CMGS
CMGS deleted the feat/parallel-volume-attach branch August 13, 2026 10:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant