microvm: auto-select chv restore mode / prepare for chv v53 - #847
Open
Benjamin Elder (BenTheElder) wants to merge 3 commits into
Open
microvm: auto-select chv restore mode / prepare for chv v53#847Benjamin Elder (BenTheElder) wants to merge 3 commits into
Benjamin Elder (BenTheElder) wants to merge 3 commits into
Conversation
Benjamin Elder (BenTheElder)
requested review from
Dmitry Berkovich (dberkov) and
Julian Gutierrez Oschmann (juli4n)
August 11, 2026 03:47
vmm.ping already answers before every boot and restore, and its reply carries the VMM's version; we were discarding it. Return it from Ping and WaitReady so callers can tell which cloud-hypervisor they are driving without running the binary again to ask. Add the classification that goes with it: cloud-hypervisor 53.0.0 made the userfaultfd restore handler background-prefault every registered page and refuse a snapshot until that finishes. The affected range is bounded rather than open-ended, so when a release stops doing it the smaller footprint of an on-demand restore comes back by moving one constant.
…hat cannot help An on-demand restore is what we want: pages fault in as the guest touches them, so an idle restored actor holds its working set rather than its whole snapshot — 16MiB against 158MiB on the counter demo. On a cloud-hypervisor that prefaults it is not merely wasteful but unusable, because the prefault starves the guest and its readiness probe never passes. Choose per VMM from the version vmm.ping reports. This is a property of the hypervisor we are driving, not something a deployment should set, so there is no knob: an unreadable version restores eagerly, which costs memory but always works, and says so. Restoring eagerly also reads every populated extent up front, so what the next snapshot captures is the whole guest and not a delta. Merging that onto the restore source copies the entire resident set to no purpose, so skip it.
An eager restore reads the whole snapshot into guest memory up front, and nothing merges against it afterwards, so the staged copy atelet left behind is dead weight: a second ~160MiB per running actor on top of the checkpoint it will go on to write. Measured on the counter demo, an actor's on-disk state halves, 318MiB to 158MiB. Remove the memory image but keep the directory and the small files beside it. atelet clears and re-stages the whole directory before any later restore, so nothing downstream depends on the image still being there.
Benjamin Elder (BenTheElder)
force-pushed
the
microvm-ch-restore-mode
branch
from
August 11, 2026 04:00
8e3c9e6 to
7c416e7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Make the micro-VM runtime choose how it restores guest RAM based on the
cloud-hypervisor it is actually driving, so we can support v53.
Why
cloud-hypervisor 53.0.0 changed on-demand restore in two ways: the userfaultfd
handler now background-prefaults every registered page (
#8150), and a snapshot isrefused while that runs (
#8556). Together these makememory_restore_mode=OnDemand— what we use today — unusable on v53: the prefault storm starves the guest and
its readiness probe never passes. The actor never comes up.
cloud-hypervisor ships minor releases and rarely backports fixes to a patch branch,
so pinning v52 means running an unmaintained hypervisor indefinitely. We need to be
able to keep up.
Restoring eagerly sidesteps both problems: it reads only the snapshot's populated
extents and registers no userfaultfd, so nothing prefaults and nothing gates a later
snapshot. But eager is the wrong choice on v52, where on-demand costs a tenth of the
memory. So this is a per-VMM decision, not a configuration knob.
How
vmm.pingalready answers before every restore and its reply carries the version;we were discarding it.
Ping/WaitReadynow return it — no extra process, noextra round trip, nothing to cache.
(
prefaultingSince/prefaultingUntil) rather than "53 and up forever", so when arelease stops prefaulting unconditionally, on-demand's smaller footprint comes back
by moving one constant.
equal: guessing on-demand on an affected version leaves the actor unable to start,
while guessing eager only costs memory.
merge (it would copy the whole resident set onto the restore source for nothing), and
drop the staged memory image (nothing pages from it or merges against it afterwards).
Measurements
kind, arm64, 2 GiB guest, counter demo. Each config measured identically, no flags set.
On v52 the runtime detects
52.0.0, selects on-demand, and reproduces the currentshipped behaviour exactly — this is a no-op on what we run today. On v53 it detects
53.0.0, selects eager, and the actor works.Within v53, the two follow-on changes are worth their own line:
Testing
vmm.pingpayloadsfrom both binaries we ship, and a test documenting how to retire the workaround.
counter continuity across five pause/resume cycles each.
go test -race ./cmd/ateom-microvm/...green; builds for linux and darwin.NOTE: This PR does not upgrade chv to v53+ by default, I'd like to close the remaining gaps first.