ci.md

Continuous integration

The GitHub Actions workflow has two microVM test layers on Azure-hosted self-hosted KVM, MSHV, and WHP virtual machines. Each backend has a pool of runners labeled by operating system, backend, and virtual-machine, plus one of two mutually exclusive roles; see Job graph and runner capacity for their sizes. Every runner but one per backend carries general, which every non-benchmark backend job requires, so those jobs can use any of them without binding a workload to a specific host. The remaining runner carries benchmark instead of general and is dedicated to the platform job. This keeps unrelated long jobs from delaying the platform measurement and makes every run measure its performance series on the same VM (see Benchmarks). Before it measures, each platform job also checks that its runner is still the host that the series' history comes from (see Expected benchmark hosts). openvmm-vmm-tests downloads the NVX guest artifacts and uses the Linux-direct kernel and Alpine initramfs to exercise OpenVMM's Linux MP-table lifecycle, TTRPC, and snapshot contracts on each backend. openvmm-unit-tests runs the OpenVMM unit and documentation tests once per operating system, on a GitHub-hosted Ubuntu runner and a WHP runner. The tests that need /dev/kvm or /dev/mshv are ignored, so the KVM and MSHV runners used to run the same 4,820 tests, and the Linux leg needs no hypervisor runner; see Job graph and runner capacity. Failed openvmm-vmm-tests jobs upload Petri's test_results directory, including guest and VMM logs, screenshots, and watchdog inspection data. These seven-day artifacts are named openvmm-vmm-tests-<os>-<backend>-<run-id>-<run-attempt>, so a successful rerun does not replace the failed attempt's diagnostics. Linux collects them from openvmm/target/vmm_tests/test_results; Windows uses <runner-temp>/<backend>/test_results. The nvx-microvm-tests-{kvm,mshv,whp} jobs consume the NVX Linux kernel and the NVX Linux kernel plus the selected Alpine or Ubuntu initramfs and exercises Linux, SMP, virtio, sandbox, and snapshot behavior through the public OpenVMM CLI. Alpine-control-only scenarios remain explicit and are rejected for the Ubuntu initramfs. Each job also boots the Azure Linux initramfs through its one-vCPU smoke set, under the same time ABI checks; the debug-kernel jobs skip it, as they skip the Ubuntu tests. Failure logs from the NVX layer are uploaded per backend. Every harness launch, in the tests and the benchmarks, scans the OpenVMM console for the guest's time ABI output, and the tests scan every virtio console they read over TCP the same way, to its end. An NVX-TIME-ABI-VIOLATION event or a failed NVX-TIME-ABI conformance line fails the scenario at once with the guest's code and detail. Guests keep the console quiet, because every console byte costs a port exit. Only the initial clock step precedes the guest's boot marker; the other boot checks finish afterwards and still power off with status 193 on failure, possibly in the middle of a scenario. A guest prints one NVX-TIME-ABI line per recorded check phase (boot, then capture and restore after a restore) and a runtime line only when /sbin/nvx-time status asks, after waiting up to 30 s for pending checks. The test runners ask after every cold boot whose shell is on a console the harness reads, the OpenVMM console or a virtio console over TCP, before any other input, and wait for the query to exit, so the console's echo of later input cannot split its lines. Cold boots with no shell there are only scanned: one-shot workloads, and the managed lifecycle, where init starts the managed agent instead of a shell. A failing boot check still powers them off with status 193, which fails the run. The query must exit 0 with a passing NVX-TIME-ABI boot line, which reports ABI version 1, generation 0, a plausible TSC rate, and the backend's LAPIC rate; a missing line means the guest image or OpenVMM does not implement the ABI, a check still pending after the guest's 30 s wait fails, and a report-only guest (NVX-TIME-REPORT) is never accepted. A guest image whose nvx-time has no status subcommand fails with that reason. The runtime line only records the wall-clock discipline's state. Benchmarks never ask, so their measured intervals stay quiet. An OpenVMM exit status of 193, 194, or 195 is reported as the guest's time ABI conformance, runtime-violation, or restore-repair power-off, together with the event that preceded it, instead of as a generic exit status. The smp, smp-snapshot, and restore-processors scenarios run the guest warp probe, /sbin/nvx-time-probe warp --bound-ns 1000, over every pair of online CPUs after each boot or restore, in two rounds with all vCPUs halted for 1 s between them, so that a host without an invariant TSC corrects the guest TSC as idle host CPUs wake (#265). Every pair in every round must stay within the time ABI's 1 µs cross-vCPU skew bound for both the backward TSC step and the ping-pong offset; a stalled pair or an inconclusive measurement also fails. smp boots each requested processor count. smp-snapshot captures one snapshot per requested count and restores it once, and the first snapshot a second time to prove that a restore leaves it reusable. restore-processors captures one boot-online CPU with capacity 8 and probes the CPUs that each restore target activated. After the 1/2/4/8-CPU restores, it restores the same snapshot once without --restore-processors. After the warp probe, every smp-snapshot and restore-processors restore runs /sbin/nvx-time status, which waits for the restore's deferred checks; it must exit 0 with a passing NVX-TIME-ABI ... phase=restore line that reports the backend's LAPIC rate, the restored CPU count, and generation 1: every scenario captures a cold-booted guest, and OpenVMM cannot capture a restored one. The query also prints the source's boot and capture lines, which keep the CPU count their checks covered, so only the restore line must match the restored CPU count. Every restore runs with OpenVMM lifecycle profiling and must report exactly one startup.vp_thread_bind record. Its startup.vp_bind_* records must show that an explicit MSHV target binds exactly VPs 0..N-1, while untargeted MSHV restores and all KVM and WHP restores bind the full capacity. Captures do not wait for a clocksource: the time ABI registers tsc at device_initcall on every backend, so the transitional tsc-early window that once let the clocksource watchdog compare tsc-early with jiffies across a restore (#253) never reaches the guest's userspace. A restore fails as soon as its guest prints NVX-RESTORE-PROCESSORS-FAIL or NVX-WARP-PROBE-FAIL, rather than waiting for the phase timeout; a failed probe also powers the guest off with status 97. Restore-processor logs record OpenVMM's time ABI rates declared event, which reports the identity MSR route, the TSC synchronization method, the native and declared TSC rates, and the rate deviation from the snapshot. The restore-downtime scenario covers the time ABI's long-downtime case. It captures four snapshots, at 1 and 8 vCPUs, each with and without rcupdate.rcu_expedited=1, then waits 30 s, longer than the guest's 21 s RCU stall timeout, and restores each one. Every restored guest must run the warp probe, report a passing restore line through nvx-time status before the harness stages its check, report /sys/kernel/rcu_stall_count as 0 two seconds later, and show an uptime of at least 30 s, which proves that monotonic time advanced by the downtime. The captures share one downtime window, so the scenario adds about a minute per backend. The time-abi-conformance scenario boots the largest requested vCPU count and runs the guest's exhaustive CI check, /sbin/nvx-time exhaustive, which the boot check leaves to CI. On every online CPU it checks every leaf 0x40000006..=0x400000ff and every base 0x40000100..=0x4000ff00 for another hypervisor signature, every identity MSR that the boot check reads, the write rules of HV_X64_MSR_TSC_INVARIANT_CONTROL, writes to the read-only identity MSRs, and reads of IA32_TSC_ADJUST and IA32_TSC_DEADLINE (checks X1 to X6). The harness requires a passing NVX-TIME-ABI-EXHAUSTIVE line for every check on every CPU, a summary with status=ok, the requested CPU count, and no failures, and exit status 0; a failure lists each failing check with the guest's detail. The guest command fits on one console line, so the console's echo of it ends before the check prints. The cpu-profile-fallback scenario covers --cpu-profile auto's fallback to a host profile on a CPU that no built-in profile serves, on runners that built-in profiles serve. It cold boots one vCPU with OpenVMM's host-cpu-unknown test hook, which makes auto treat the host CPU as unknown, and requires exactly one NVX-CPU-PROFILE-FALLBACK: warning that names a host profile with its digest and links the CPU profile request form, a cold boot that declares that profile, and passing boot checks. It then captures the guest and restores it with auto and no hook: the restore must take the snapshot's host profile without another warning, and pass the warp probe and its restore checks. The snapshot-core scenario first sends a snapshot request to a guest that OpenVMM launched without a snapshot destination. OpenVMM releases the request, so it returns in the source, which must continue exactly once. Before the request, the guest's snapshot agent saved and overrode the stall detectors' settings, and it must restore them when the request returns (nvx-time cancel-capture). Afterwards, nvx-time status must exit 0 with a passing boot line at generation 0 and no restore line, and /sys/module/rcupdate/parameters/rcu_cpu_stall_suppress must read 0.

At the end of a passing run, test-microvm prints one NVX-TIME-ABI-EVIDENCE: line and adds it to the GitHub job summary, because CI keeps the guest logs only for failed jobs: the number of warp probe runs with their worst max_abs_offset_ns and max_backward_ns, the count and elapsed_us range of the newest check that each nvx-time status query reported (boot after a cold boot, restore after a restore, and capture after snapshot-core's released request), the exhaustive check's summary, and the restore-downtime stall counts. Where the guest reports a check's CPU time (cpu_us), the line also gives its range and how many checks exceed the backend's CPU-time budget for their phase. The budgets come from guest measurements and exempt the first capture after a rolled-back capture in the same VM process; CI never makes one, because each VM process sends OpenVMM at most one snapshot request and a failed capture fails its scenario. CHECK_CPU_BUDGET_US in scripts/nvx_tools/time_abi.py holds the budgets, one per backend and phase. elapsed_us is wall time, including waits behind the workload, and has no budget. CI gates on neither: the performance gate is the A/B comparison outside CI.

The nvx-microvm-debug-{kvm,mshv,whp} jobs run test-microvm --debug-kernel on the CI debug kernel (build/vmlinux-debug, built from kernel/config-microvm-debug), whose soft-lockup and hung-task detectors production kernels leave out. It selects the same-host restore scenarios smp, smp-snapshot, restore-processors, restore-downtime, and snapshot-tiers. The jobs skip every other test step, because those boot the production kernel, which the nvx-microvm-tests-{kvm,mshv,whp} jobs already cover: the public managed execution configuration, the Ubuntu and Azure Linux guest tests, the Ubuntu sandbox layer and live-share smoke tests, the caller-owned live-share test, and the aci_edge_sandboxes lifecycle test. Any RCU stall, soft lockup, or hung task makes the guest's time ABI watcher power off with status 194, which fails the run. Every guest those scenarios restore asks nvx-time status before it exits (snapshot-tiers after its tier assertions), which waits for the restore's deferred checks and must report a passing restore line, so no restore leaves its checks pending. The one exception is snapshot-tiers' gate-timeout check, whose guest OpenVMM stops at the restore gate on purpose. The harness refuses a kernel whose vmlinux-debug.config lacks the detectors, because the guest's boot check of those detectors passes vacuously. To bound the cost, pull requests run the debug kernel on KVM only, while dev pushes and full runs run it on every backend; each job takes about five minutes on its own runner, in parallel with the other microVM jobs. The jobs gate the required status check, the development release, and performance persistence. The GitHub-hosted debug-kernel job builds the debug kernel beside the shared artifacts job (build-guest-artifacts with guest-images: "false") and caches it under its own key, so a kernel rebuild delays only the debug jobs, and a failed debug kernel build fails the required status check and blocks the release.

Every job that uses the validate-runner action first reports the Linux runner's kernel, CPU, clocksource, and TSC flags. It then qualifies the runner for the time ABI with nvx.py doctor --checks H1 H2 H4 --ci-schedule (see Host qualification): the backend, the CPU fingerprint and generation, and the TSC rate stability on its short schedule. The microVM and platform jobs run it after downloading OpenVMM and add OpenVMM's CPU profile check (H2) and preflight (H3), and keep the fingerprint as an artifact when the profile check fails; the other jobs pass --no-openvmm. This takes a few seconds. It reports the CPU generation, the built-in CPU profile that auto selects, and the measured TSC rate in the log and the job summary, and fails the job with a stable code when the runner is not qualified, for example E_PROFILE_HOST_UNKNOWN on an unknown CPU generation, where auto would fall back to a host profile, which CI never uses. The platform jobs also run the guest warp probe (H6) on CI's schedule, which adds about 25 s, because their benchmarks boot guests without the microVM scenarios' probes. The doctor gates only on measured properties, alike on every backend: it records the host OS's invariant-TSC flags and clocksource as evidence, and the guest warp probe measures the skew that a host without an invariant TSC causes. Before the time ABI, never-restored guests on such an MSHV runner VM, an Ice Lake 8370C, hit cross-vCPU TSC warps when an idle host CPU woke (#211, #265), which is why the probe schedules include idle gaps. Under the time ABI, its guests' warp probes measure at most 36 ns, so validate-runner only notes a missing nonstop_tsc, and the Linux runner setup script only warns. The OpenVMM vmm-tests run no warp probe: they boot OpenVMM's own test guests, whose verdicts don't depend on the skew bound. Runner labels do not encode the generation; per-PR CI captures and restores on one runner, so generations never mix.

The warp probe replaced the restore-tsc-sync scenario, its test-only clearcpuid=tsc_adjust kernel option, the guest's scan of the kernel log for TSC warp and instability messages, and the fresh-boot TSC control that classified those failures (#211, #265). Under the time ABI the guest's TSC is tsc_reliable, so Linux skips its CPU-online warp check and never logs the messages that guard looked for; the probe measures the 1 µs bound on every CPU pair instead.

The console-exit scenario delays host console reads for two seconds after snapshot restore to exercise output backpressure. For each requested processor count it requires byte-exact delivery of a 64 KiB payload and the final marker, and preserves guest exit statuses 0 and 37. This checks both device and host-relay draining without adding sleeps to the measured benchmark workloads. The harness waits for the output reader's EOF notification even after the process exits, so delayed final output chunks cannot create a false failure.

The command-line-limit scenario covers the x86 guest kernel's 2,048-byte command-line buffer. OpenVMM appends its own tokens after the caller's, ending with the network bootstrap, so they are the first that the kernel would cut from a command line that doesn't fit, and OpenVMM refuses an effective command line longer than 2,047 bytes instead. With a NIC attached, the scenario boots once with a one-byte padding token to learn the effective command line from the guest, then pads the caller's tokens until it holds 2,047 bytes. The guest's /proc/cmdline must equal the first one with the longer padding, and the guest must have installed the IPv6 default route that the last token, virtnet_gw6=, names. OpenVMM must refuse a command line one byte longer before boot, with an error that names its length and the limit.

Shared guest artifacts are built with Docker on a GitHub-hosted Ubuntu runner. The kernel, Alpine initramfs, Ubuntu initramfs, and Ubuntu EROFS layer use separate cache keys. Every guest image key includes the shared binary-build modules, the Dockerfile, and .dockerignore; the Rust builder image itself is pinned by digest. Every guest image key also ends with the path-aware digest of the Rust guest binaries' inputs that the guest package manifests record, which nvx.py guest-binary-source-identity prints, because the images carry only those binaries of nvx-rs and hashFiles hashes contents only. The inputs are the files that the guest binaries' build reads: rust-toolchain.toml; the crate's manifest, lockfile, build script, and Cargo configuration; every file under nvx-rs/src/ but the host side, which is the host and testing modules and the nvx-rs and nvx-fake-openvmm binaries, and the test modules, which are the tests.rs files, the tests directories, and the managed agent's loopback harness; and the two Python modules that define the build. GuestRustBuildConstants in build_constants.py selects them, and every guest build fails when Cargo's dep-info for the nine executables lists another file, such as a host module that guest code starts to compile. A change to the host side or a test of nvx-rs therefore rebuilds no guest image, and a change to any file that a guest binary compiles rebuilds them all. Ubuntu keys also include the Canonical archive pin, supplemental package lock, common guest sources, shared download and guest descriptor modules, and converter implementation. Artifact upload retains the Alpine filenames and adds the distinct Ubuntu filenames. Each backend also boots the Ubuntu initramfs and runs /sbin/nvx-sandbox-smoke from the Ubuntu EROFS layer as UID/GID 65534 over a fresh ext4 scratch copy, with a 32 MiB --memory-max and an 8-process --pids-max. The probe verifies the sandbox security profile: the fixed identity, empty capability sets and no_new_privs, private mount, PID, and UTS namespaces, the private device view and read-only sysfs, the container cgroup, the overlay root over read-only image layers with its upper directory on scratch, and that the memory controller kills an allocation of twice the limit and the pids controller stops the workload at eight processes. The live-share runs below repeat the profile checks. Each backend also runs the public managed-exec-config and sandbox-lifecycle scenarios of test-microvm, which drive nvx.py sandbox provision, start, exec, stop, and deprovision as subprocesses (see Run): request-scoped working directories and environments, timeouts, nonzero exits, stdout and stderr, bounded output and outcome reports, warm-guest state across requests and restarts, identity refusal, fail-closed lifecycle transitions, a crashed OpenVMM's stale runtime state, and teardown that leaves no OpenVMM process, control endpoint, capability, or runtime record while the layer and scratch artifacts survive. Their bounded evidence is part of the backend's failure logs. The same entrypoint then verifies a live virtio-fs share inside the container: a read-write /workspace share with a denied subdirectory must round-trip guest writes to the host, and a read-only /opt/hostedtoolcache share must reject writes and symbolic links. In the read-write share, the guest also exercises the symbolic-link primitives that package managers use: it creates an npm-style relative .bin link to an executable and runs it through the link, renames, replaces, and removes links, and creates absolute, dangling, outside-the-share, and denied-path links. The host then checks that every target is preserved exactly (on Windows, by decoding each WSL-style reparse point, which Windows does not follow) and that no link read or modified host data outside the share or in the denied path. A managed sandbox then repeats the read-write check through provision, start, exec, and stop, and must report a successful outcome with a cleanly unmounted scratch filesystem, which shows that stop unmounted the share and overlay first. Each backend then attaches a read-write /workspace share, a read-only /opt/hostedtoolcache share, a read-only /config/settings.json file, and a read-write /results/output.txt file to one sandbox at the same time, as the children of one aggregate virtio-fs device; each directory has its own denied subdirectory, and each file lies beside a file that no share exposes. The smoke script runs the read-write and read-only checks on the two shares and verifies that a link in the read-write share cannot write into the read-only share. It requires each file to be a virtiofs bind mount with its mode, alone in its guest directory, the read-only file to refuse writes with EROFS, and the workload to rewrite the read-write file but neither remove nor rename either one. The host requires the guest's writes in the workspace and the read-write file, an unchanged tool cache, read-only file, and denied files, and unchanged files beside the shared files, both for a one-shot run and for a managed sandbox whose format-6 configuration must persist every share and its kind and whose stop must leave a cleanly unmounted scratch filesystem. Each backend then attaches a read-write /workspace share whose access policy denies nvx-denied, allows nvx-denied/nvx-allowed, and makes only nvx-writable writable, with every host directory and file writable by every identity so that only OpenVMM can refuse a write. The smoke script's policy mode requires the workload's writes to succeed only in nvx-writable and to fail with EROFS everywhere else, a hard link of a read-only file into nvx-writable to fail, and nvx-denied to list only nvx-allowed, whose file stays readable while nvx-denied/secret stays hidden. The host requires the guest's writes in nvx-writable and no other change to the share, both for a one-shot run and for a managed sandbox whose format-5 configuration must persist the denied, allowed, and writable paths and whose stop must leave a cleanly unmounted scratch filesystem. The default filesystem-policy scenario of test-microvm covers the same policy as guest root, whose permission checks mode bits never stop: every mutation outside the writable paths, through a second mount of the tag as well, fails with EROFS, a hard link fails with EXDEV, a denied path nested inside an allowed path stays hidden, invalid policies are rejected before boot, and a snapshot restore requires the same policy and keeps a writable handle open across the capture. On Linux, a world-writable share then runs with --mount-owner caller. The runners grant OpenVMM neither CAP_SETUID nor CAP_SETGID, so it can neither assume the 65534 workload identity nor drop the backend group through which it opens /dev/kvm or /dev/mshv. The smoke script's eperm mode therefore requires reading and listing the share to fail with EPERM and writes to fail, and the host requires the share to be unchanged, both for a one-shot run and for a managed sandbox whose configuration must persist the mode. The filesystem-owner scenario of test-microvm likewise requires guest root and a foreign guest identity to fail with EPERM on the runners, and a root-owned export to be rejected before boot; on Windows, --mount-owner caller must be rejected before boot. The OpenVMM unit tests check the host identity that each request runs as, including guest root squashed to an export owner other than OpenVMM. Where OpenVMM can drop its supplementary groups, the scenario instead requires files that guest root creates to belong to the share owner rather than to root, and squashed root to fail to chown a file, give it another of OpenVMM's groups, or create a device node. When the test process also passes CAP_SETUID and CAP_SETGID to OpenVMM, as on a privileged developer host, the scenario requires the foreign identity to own the files it creates. Linux/KVM runs the broader Ubuntu SMP, managed lifecycle, network snapshot, blockless snapshot, and workload-identity set.

OpenVMM release executables and provenance are built once by the independently addressable build-openvmm-linux-gnu, build-openvmm-linux-musl, and build-openvmm-windows-msvc producer jobs. KVM workloads and MSHV microVM tests consume the GNU artifact, MSHV platform workloads consume the musl artifact, and WHP workloads consume the Windows MSVC artifact. Each workload can start after its compatible OpenVMM producer and the shared guest-artifact job finish, without waiting for unrelated OpenVMM targets.

All three producers call the same Python build workflow, passing the validated runner backend explicitly through build-openvmm --backend. The backend is carried in OpenVmmBuildConfig; the build workflow maps KVM, MSHV, or WHP to GNU, musl, or MSVC without probing runtime devices. CI therefore retains its musl build for MSHV without maintaining a separate shell build path.

The kernel, debug kernel, and initramfs cache keys include docker/Dockerfile, which pins the build image and its toolchain and in which every guest artifact builds, .dockerignore, which Docker applies to the build context before COPY, and build_config.py and build_constants.py, so a toolchain, shared build configuration, or constant change invalidates cached guest artifacts and their provenance. The Ubuntu distro layer shares the Ubuntu input hash.

The producer handoff uses one-day workflow artifacts rather than caches. Each consumer downloads both the normalized executable and its build provenance, then restores executable permissions on Linux.

nvx.py runs nvx-rs for its sandbox command and for the egress-policy and artifact steps of its run command, so the microVM test jobs take nvx-rs the same way, and every release package ships it. Each NVX CLI job builds it for its operating system with --release --locked --features cli, puts it at nvx-rs/target/release/, where nvx.py looks for it, and runs the CLI tests with it, with NVX_RS naming it for the tests that run it directly: nvx-cli-linux builds a static x86_64-unknown-linux-musl executable, which runs on every self-hosted Linux runner, and nvx-cli-windows an x86_64-pc-windows-msvc executable with a static C runtime, which needs no Visual C++ redistributable. Each job records the build's provenance with record-nvx-rs-provenance, and, once the tests pass, uploads the executable and the provenance as the one-day nvx-rs-linux-musl-executable and nvx-rs-linux-musl-provenance artifacts, or the nvx-rs-windows-msvc-* pair. The workflows that take them name the pair by its base name, nvx-rs-linux-musl or nvx-rs-windows-msvc, as they name the OpenVMM artifacts. Each microVM test job of that operating system downloads the executable into nvx-rs/target/release/ and restores its executable permission on Linux. The platform jobs run neither nvx.py sandbox nor nvx.py run, but their package-release step downloads both, the provenance into build/, so that each package ships the nvx-rs of its operating system. The managed-lifecycle scenario also builds and runs the ignored nvx-rs test tests/managed_lifecycle.rs with Cargo on the runner, as test-aci-edge-sandboxes builds the tests of its crate.

Once the required artifacts are ready, benchmarks run in parallel with the NVX test layer, each on its backend's benchmark runner. All three use virtual-machine performance series and the constrained eight-CPU affinity policy. Development releases and performance baseline updates still require every applicable test and benchmark lane to pass. The microVM correctness jobs gate every use of benchmark results, so the benchmark action runs no correctness scenario of its own (#286). Required status check requires each nvx-microvm-tests-* job that the change schedules to succeed. Publish development release and Persist performance baseline run only on dev pushes in which every microVM test job succeeded or was skipped. The pull-request Performance regression gate reads only the platform jobs' results and publishes nothing. The counting LAPIC that the benchmarks depend on is covered by the smp scenario of those jobs. smp-lapic repeats smp with the counting-LAPIC facts asserted, for explicit local use only; no CI job runs it. Benchmarks lists the jobs that consume, gate, publish, or persist the results. The workflow uses the read-only OpenVMM deploy key stored in the OPENVMM_DEPLOY_KEY Actions secret to fetch the private submodule at its pinned commit. Shared guest binaries and development release packages move through short-lived workflow artifacts alongside the OpenVMM handoff and benchmark results. Caches only accelerate reproducible build inputs and outputs; consumers do not depend on them as a handoff. Pull requests gate regressions against recent matching-platform history, and successful pushes to dev append their p50 values under data/. A pull request that changes no input of the platform jobs skips them and the gate (see Change classification). Metadata-only performance jobs use GitHub-hosted Ubuntu runners. Provisioning instructions are in the runner bootstrap guide.

Persistent runners accept pushes, scheduled and manually dispatched full runs, and same-repository pull requests only. Fork pull requests run the GitHub-hosted validation jobs but do not execute code on the Azure runner fleet. A maintainer must stage an external contribution on a trusted repository branch before running the backend matrices.

Job graph and runner capacity

A dev push, a pull request that is ready for review, or a scheduled or manually dispatched full run starts these jobs at once: quality, aci-edge-sandboxes, nvx-rs, nvx-cli-linux, and nvx-cli-windows on GitHub-hosted runners, and openvmm-changes, which classifies a push or pull request and selects every suite for a full run (see Change classification). When it finishes, the jobs that it selects start: artifacts and debug-kernel restore or build the guest artifacts on GitHub-hosted runners, the three build-openvmm-* producers restore or build OpenVMM on a KVM, an MSHV, and a WHP runner, and openvmm-unit-tests starts on a GitHub-hosted and a WHP runner. openvmm-vmm-tests waits for artifacts. Each microVM test and platform job waits for artifacts, its OpenVMM producer, and the NVX CLI job of its operating system, and the debug-kernel jobs also wait for debug-kernel. Performance regression gate, Publish development release, Persist performance baseline, and Required status check wait for the jobs whose results they check. Required handoffs between these jobs use same-run workflow artifacts; caches only skip rebuilding their inputs.

The NVX CLI tests do not depend on the backend, so they run once per operating system in nvx-cli-linux and nvx-cli-windows rather than in every hypervisor job. Each checks out OpenVMM, whose pinned CPU profiles a test reads, and uses Python 3.12, the release that the self-hosted runners provide; the Windows job also installs the pinned Rust release with the minimal profile for the curl shim test and the nvx-rs build. Each job first builds the nvx-rs executable that nvx.py runs, tests with it, and hands it to the microVM test and platform jobs of its operating system, as described above. Fork pull requests lack the OpenVMM deploy key, so the CLI jobs run, like the hypervisor jobs, only for pushes, full runs, and same-repository pull requests. Because the microVM and platform jobs of each operating system wait for its CLI job, a broken CLI does not occupy the hypervisor runners. Required status check, the development release, and performance persistence require both CLI jobs.

The KVM and MSHV pools have five runners each, and the WHP pool has nine. In each pool, one runner carries benchmark and runs only the platform job, and the others carry general and run all non-benchmark backend jobs. Platform jobs from overlapping workflow runs still measure one after another on their backend's single benchmark runner, but they never wait behind an unrelated build or test.

The role split follows an October 15-20, 2026 scheduling audit, when each pool had five runners. Before the split, the benchmark runners spent about one third of their assignments on non-platform work; azure-windows-5, for example, ran 102 platform jobs and 62 build or test jobs. Platform jobs queued for p50 6.3, p90 22.2, and at most 34.1 minutes on WHP, for p90 15.7 and at most 21.1 minutes on KVM, and for p90 11.2 and at most 18.7 minutes on MSHV. The WHP platform job finished last in about a third of dev pushes and nearly half of full pull-request runs. Replaying the observed arrivals and durations with dedicated benchmark runners reduced WHP platform wait to about p90 3.6 and at most 8.4 minutes.

The WHP jobs are the longest: the OpenVMM VMM and unit tests compile OpenVMM on Windows for about 15 and 13 minutes before testing, so overlapping runs queue on WHP first. Dedicating the benchmark runner takes it out of its pool's general capacity. In the replay, which left four general WHP runners, the WHP non-benchmark p90 wait rose from about three to five minutes, while the KVM and MSHV non-benchmark waits stayed near zero. The WHP pool has since grown to eight general runners, more than the five that served these jobs before the split. A change that adds a job or a shard needs general runners to run it in parallel; otherwise it only queues.

The OpenVMM unit tests need no hypervisor. In October 2026 (#697), on GitHub-hosted runners, which have four vCPUs and keep no sccache, the Linux suite took 12 to 18 minutes from scratch on four runners, against 6.5 on an MSHV runner, and the Windows suite took 32 minutes from scratch and 21 with a warm sccache, against 15 on a WHP runner. Every hosted run passed the same 5,294 Linux or 4,933 Windows tests as the self-hosted runners. The Linux leg therefore runs on ubuntu-latest: even at its slowest, it would have finished before the rest of the run in each of the 208 successful runs that ran the suite from October 8 to 21, and it leaves about 900 MSHV runner-minutes a week, about 15% of that pool's busy time, to the jobs that need a hypervisor. It installs the pinned Rust release and cargo-nextest, keeps Cargo's home in the job's temporary directory, as the self-hosted runners do, and skips validate-runner and sccache, which only they have. The Windows leg stays on WHP: a GitHub-hosted runner would have delayed 69% of the runs that ran it, and still 13% with a warm sccache, which would take over 2 GiB of the repository's 10 GB Actions cache, already full. Larger GitHub-hosted runners, which public repositories pay for, or a self-hosted Windows runner without a hypervisor would cost about as much as another general WHP runner.

Run concurrency

Each pull request runs in a concurrency group of its own, and a push to it cancels the run of the previous head, whether that run waits or runs. Pushes to dev share one group and run one at a time in the order in which they arrive. The group keeps up to 100 waiting runs (queue: max), so a merge does not cancel the waiting run of an earlier merge. Until October 2026 the group kept a single waiting run, which each newer push replaced: 12 of the 199 dev pushes from October 4 to 21 lost their run that way, five of them merged within four minutes on October 20. Those commits got no development release and no performance results, and because the detector compares a push with its previous commit, the run that followed classified only its own changes. GitHub rejects queue: max together with cancel-in-progress: true, so the workflow derives both settings from the event. Each scheduled or manually dispatched full run has a group of its own (see Change classification).

A queued run waits for the runs ahead of it. Replaying the dev pushes of October 4 to 21 one at a time, at their measured durations and the median one for those that were cancelled, half would have started at once, one in ten would have waited at least 50 minutes, and the longest wait, on October 20, would have been about three hours.

Change classification

openvmm-changes lists the files that changed between the pull request's base and head, or between a push's previous and new commits, and nvx.py classify-ci-changes selects the suites from their paths. The detector uses git diff --name-only --no-renames -z and the classifier's --null option so both sides of a rename and unquoted filenames, including embedded newlines, participate in the decision. Documentation, every path under doc/ or ending in .md, selects no suite, even in a suite's input directory, such as scripts/setup/README.md, because no suite builds, boots, or runs it. Every other path selects each suite that takes it as an input:

• run-microvm-tests enables the NVX microVM tests and their debug-kernel jobs. Their inputs are the OpenVMM pin, the guests and every input of their cache keys, the nvx crate, whose executable their scenarios run and whose guest binaries every guest ships, the aci_edge_sandboxes crate and the runtime adapter in tests/runtime-adapter/, which they test, the workflows, actions, and CLI modules that their jobs and the jobs that build their artifacts run, the runner setup scripts, the Rust pin, and .gitattributes, which checks out their scripts on Windows.
• run-benchmarks enables the platform jobs, which benchmark each backend and package its release, and so a pull request's performance regression gate. Their inputs are the OpenVMM pin, the guests and every input of their cache keys, the nvx crate, which every package ships, the performance history and expected hosts under data/, VERSION, LICENSE, the workflows, actions, and CLI modules that the platform jobs, the gate, and the jobs that build their artifacts run, the runner setup scripts, the Rust pin, and .gitattributes. A pull request sets it only when one of these inputs changed, but a push sets it for every change but documentation; see Development releases and performance history.
• run-openvmm-unit-tests and run-openvmm-vmm-tests enable the OpenVMM suites. A push or pull request sets each only when one of that suite's own inputs changed. The unit tests depend on the OpenVMM pin, the Rust pin, the runner setup scripts, ci.yml, the actions that their job uses, nvx.py, and the modules that ci.py imports. The VMM tests also boot the kernel and Alpine initramfs, so they add every input of those artifacts' cache keys and the build-guest-artifacts action. Of nvx-rs, those are only the inputs of the guest binaries, which GuestRustBuildConstants.SOURCE_PATTERN matches, so a change to its host side or tests runs the NVX microVM tests and the platform jobs but not the VMM tests.
• run-guest-artifacts enables artifacts and debug-kernel, which build the guest artifacts and the CI debug kernel, and run-openvmm-binaries the three build-openvmm-* producers. Neither has a use of its own, so the classifier sets each exactly when a job that uses its artifacts runs: the guest artifacts for the OpenVMM VMM tests, the NVX microVM tests, or the platform jobs, and the OpenVMM binaries for the microVM tests or the platform jobs. Every input of an artifact is an input of each job that uses it. The debug kernel is built with the guest artifacts, as before, because the development release and performance persistence require its build too.

Every CLI module is an input of the microVM tests, and of the platform jobs, unless none of that suite's jobs runs it, so a new module selects both until the classifier excludes it. microvm_tests.py and the modules that only it, nvx.py sandbox, or nvx.py test-aci-edge-sandboxes use, with the microvm_test_scripts/ that it runs in guests, are inputs of the microVM tests alone, and performance.py is an input of the platform jobs alone. The modules that both kinds of job run, such as benchmark.py, which microvm_tests.py imports, doctor.py, which every runner validation runs, and release.py, whose verify command the OpenVMM builds run, select both.

A push or pull request that changes neither OpenVMM suite's inputs, such as one that changes only the microVM tests, the benchmarks, the Ubuntu or Azure Linux guests, or the host side of nvx-rs, therefore skips the two jobs that occupy the WHP runners longest. A pull request that changes only the microVM tests, such as their scenarios, the aci_edge_sandboxes crate, or the runtime adapter, also skips the platform jobs, which wait for their pool's single benchmark runner, and one that changes only the performance tooling, such as performance.py, the run-benchmark action, or data/, skips the microVM tests. One that changes the input of no suite, such as the CLI tests, the adversarial campaigns, or the GitHub-hosted checks, runs only the GitHub-hosted jobs.

The nightly schedule runs at 05:17 UTC, and a manual workflow dispatch starts the same full run. Both select the full matrix before the detector examines a comparison base. Each full run has a separate concurrency key, so it neither cancels nor queues behind a dev push. Full runs build, test, benchmark, and package, but publish and persist nothing and save no caches. Only dev pushes save the OpenVMM caches, and only pushes and pull requests save the guest artifact and pip caches, through the save-cache input of the build-guest-artifacts and check-quality actions. A full run still builds the guest artifacts that miss the cache, but installs the Python tools without the pip cache, which setup-python saves whenever it misses. A failed nightly run opens or updates the nightly failure report.

For a push or pull request, the detector runs the full matrix instead when the comparison base or a commit is unavailable, when the diff or the classifier fails, or when the classifier reports an output twice or not at all, because a missing output would skip its jobs. Required status check passes the same outputs to check-required-ci, which expects every scheduled job to succeed and every other job to be skipped. As in ci.yml, it schedules a job only when the jobs that build the artifacts that it uses are scheduled too. Tests in scripts/test_nvx_tools.py derive each suite's inputs and fail when a pattern misses one. The OpenVMM suites' come from the actions that their jobs use, the modules that ci.py imports, and the guest cache keys, including the files of the guest binaries' source identity. The microVM tests' and the platform jobs' come from those jobs and the jobs that build the artifacts that they use: the reusable workflows and actions that these reach, the files that the cache keys there hash and the source identity's, the CLI modules of the nvx.py commands that they run, which the tests map from nvx.py's parser, with every module that those import and every file that those read beside them, and the nvx crate, from which the NVX CLI jobs build the executable. The tests also check that no other file of nvx-rs selects the VMM tests or changes a guest image key, while every one selects the NVX microVM tests and the platform jobs.

Development releases and performance history

A dev push publishes its commit's development release and appends the platform jobs' results to the performance history. Publish development release and Persist performance baseline require all three platform jobs to succeed, so a push that skipped them would publish nothing partial or stale: it would publish no release and record no history. Its commit would have no release of its own, though, and download would install an ancestor's, which is older still when that ancestor's release failed. A push therefore runs the platform jobs, and the jobs that build their artifacts, for every change but documentation, as every push did before the classifier selected the platform jobs apart from the tests. Of the 108 merges to dev from October 10 to 21, 2026 that changed more than documentation, 10 changed no input of the platform jobs, and each of their pushes spent about 19 benchmark-runner minutes on them: the KVM, MSHV, and WHP platform jobs take a median of 5.3, 5.5, and 8.6 minutes. Pull requests publish and persist nothing, so they run the platform jobs only when one of their inputs changed.

Nightly failure reports

GitHub notifies only one person when a scheduled run fails: whoever last changed the workflow's schedule or enabled the workflow again. report-nightly-ci reports a completed nightly run in one tracking issue instead, the nightly failure report, which a hidden marker in its body identifies. Only an issue that GitHub Actions or someone with a role in the repository opened counts, so a copy of the marker cannot take the report over. A failing run opens the report, or updates it when one is open, and a passing run closes every open report. The report shows the latest failing run, and a table of every failing run attempt since it opened, newest first, which its body also records in a hidden state comment. Every later failure also adds a comment with its details, which notifies the issue's participants, such as its assignees. Reporting an attempt again only refreshes the body, while a rerun's attempt counts as another failure. The command reports the latest nightly result, which a later run or attempt than the one that triggered it may hold: the newest scheduled run's latest completed attempt, which is the previous attempt while a rerun is in progress. It skips a result older than the newest run that the report records. A late result therefore cannot reopen, rewrite, or close the report. When it finds several open reports, it updates the oldest and closes the others as duplicates.

The nightly-ci-report.yml workflow runs the command whenever a CI run on dev completes, so a failing nightly run is reported within minutes of its last job. Its workflow_run trigger reacts to the run rather than to its jobs, so it also reports a run that was cancelled or whose jobs never started, and it adds nothing to ci.yml. Its job runs only for scheduled runs: a dev push's run skips it, pull request runs, whose branches the trigger excludes, never start the workflow, and a manually dispatched run notifies the person who started it. The job checks out the default branch, never the run's code. Its token can read the repository, the run's jobs and logs, and their check annotations, and write issues, and no other workflow can write issues. The job runs one report at a time; a newer reporter can replace one that waits for its turn, but it reports the latest nightly result anyway. Its job summary repeats the report.

The report links every job that failed, timed out, or was cancelled, names its failing step, and quotes its evidence: the step's own error annotation, or else the last error line of its log, or, without the log, the job's most specific annotation. Up to 15 lines of that log, without the echoed commands, follow in a collapsed block. The command reads annotations and logs for at most four minutes, and classifies the remaining jobs from their steps alone. Each job counts in one group:

• Infrastructure failures: GitHub reports a lost or stopped runner, or a job that no runner picked up; a setup step failed or timed out, such as Set up job, a checkout, Validate runner, a download, an upload, a tool installation, or a cache step; the benchmark step reports that its runner is no longer the host of its series' history (see Expected benchmark hosts); or the failing step's log shows a network or download error, such as an unresolved name, a reset connection, a rate limit, or a server error, or a full disk, other than in a warning or a notice of a retry, which the retry may have overcome. Only GitHub's runner errors count for the whole job, so that a later step, such as an upload of diagnostics that fails on the network, cannot hide a test failure.
• Test and build failures: every other job that failed or timed out. Without positive evidence of the infrastructure, a failure counts here, so that the report never dismisses a regression; the curl and wget errors of the microVM tests' guest network checks count here too.
• Cancelled jobs: jobs that stopped because their run was cancelled.

Required status check fails whenever another job fails, so the report lists it apart, unless it fails alone, for example because a job that should have run was skipped. A run that fails without a failing job, as when GitHub cannot start it, is reported with no jobs.

The workflow assigns each new report to its triage owner, the CI maintainer ppenna by default, and labels it bug, which the repository defines. The owner triages a new report within a working day:

1. Open the failed jobs of the latest failing run. For an infrastructure failure, remove its cause, then rerun the run's failed jobs; a rerun that passes closes the report. A runner that fails Validate runner needs the runner setup again, and one that stays offline needs its virtual machine restarted or replaced.
2. For a test or build failure, find the change that caused it, typically one that merged since the last passing nightly run without running the failing suite, and open an issue or a pull request that links the report.
3. Leave the report open until a nightly run passes, which closes it. Close it by hand only when nothing needs a fix, as after a deliberate cancellation.

To hand triage over or to rotate it, change the workflow's --assignee options, and reassign the open report.

Host qualification

python3 scripts/nvx.py doctor --backend <kvm|mshv|whp> qualifies a host for the time ABI. It runs checks H1 to H7 in order, prints one NVX-DOCTOR: check=<id> status=<pass|fail> detail="..." line per check, and exits with status 1 if any check fails. --checks selects a subset, and --summary appends a Markdown table with the CPU generation, profile, rates, and skew metrics to a file such as $GITHUB_STEP_SUMMARY. H4 and H6 run on long qualification schedules, which take about three minutes; --ci-schedule selects CI's short ones. A failure that matches a time ABI failure code starts its detail with the code in brackets, for example [E_PROFILE_HOST_UNKNOWN]. A check fails, and never passes, when it can't read a fact it gates on: missing or unrecognized tool output fails it, and an error in one check fails only that check. Only evidence fields, such as the microcode and the invariant-TSC flags, may read unknown.

Check	Implementation
H1	/dev/kvm or /dev/mshv is readable and writable, and a KVM host has no /dev/mshv; on Windows, WHvGetCapability reports a hypervisor
H2	Vendor, family, model, stepping, microcode, and OS build from /proc/cpuinfo or the Windows registry. Then openvmm --hypervisor <backend> --cpu-fingerprint <path> writes the host's CPU fingerprint (by default nvx-cpu-fingerprint-<backend>.json in the probe directory; --cpu-fingerprint overrides it), checks it against the pinned profile that auto selects from OpenVMM's catalog, which shares one profile per generation across backends, and prints one NVX-CPU-PROFILE: line. The check never falls back to a host profile, so a CPU on which auto falls back fails H2. H2 reports the generation and the profile from that line, so a profile that an OpenVMM pin adds qualifies its hosts without an NVX change. H2 requires exit status 0, status=pass, the same backend, and a catalog profile of the reported generation, reports the profile and surface digests, and otherwise fails with OpenVMM's code, for example [E_PROFILE_HOST_UNKNOWN] on a CPU that no profile serves or [E_PROFILE_UNSUPPORTED] naming every unsupported CPUID bit. On MSHV and WHP, OpenVMM checks the entries outside the profile on a probe partition configured from the profile, as a cold boot does. With --no-openvmm, H2 maps the host with NVX's copy of the catalog instead: skylake-sp (6/85, steppings 0 to 4, intel.skylake-sp.v1), icelake-sp (6/106, intel.icelake-sp.v1), emeraldrapids (6/207, intel.emeraldrapids.v1), alderlake (6/151 and 6/154, intel.alderlake.v1), milan (AMD 25/1, amd.milan.v1), genoa (AMD 25/17, amd.genoa.v1), or turin (AMD 26/2, amd.turin.v1). Any other CPU, including Cascade Lake and Cooper Lake, fails with E_PROFILE_HOST_UNKNOWN. A unit test keeps the copy equal to the profiles of the OpenVMM submodule's pinned revision, which it reads from the gitlink's commit, so the NVX CLI jobs validate the CLI after they check out OpenVMM. The host OS's invariant-TSC flags (constant_tsc nonstop_tsc, or the CPUID bit on Windows) are recorded as evidence and never fail the check, because they don't decide what a guest observes: Azure WHP hosts show the CPUID bit but cannot offer invariant TSC to partitions, and their guests measure tens of nanoseconds of skew
H3	openvmm --x-time-abi-verify builds the partition and runs the time ABI preflight without running the guest. Its NVX-TIME-ABI-VERIFY: line must report status=ok for the backend, plausible declared and native TSC rates, the backend's LAPIC rate, and a cpu_profile that is a catalog profile, <vendor>.<generation>.v<revision> of any generation but a host profile's host, which rejects the host profile that auto falls back to on a CPU that no built-in profile serves, and, when H2 runs too, a revision of the profile H2 names. A failed preflight reports OpenVMM's code, for example [E_TSC_SYNC_UNSUPPORTED]
H4	Samples of the TSC against the host's monotonic clocks, with sleeps between them so the host's CPUs idle: 13 samples 10 s apart, or 3 samples 1 s apart with --ci-schedule. Each sample reads the TSC between two reads of a clock, keeping the tightest of 64 brackets; its uncertainty is half the bracket plus half the clock's resolution. A clock that returns the same value to consecutive reads is coarser than one read, so the probe takes its smallest step as its resolution: Hyper-V's reference TSC page advances the Linux clocks in 100 ns steps although clock_getres reports 1 ns. The interval stability is judged against a clock that time synchronization never steers, CLOCK_MONOTONIC_RAW on Linux and QueryPerformanceCounter on Windows: every interval between consecutive samples must be conclusive within 0.25 ppm, and the interval rates must agree within 1 ppm. chrony's frequency updates move CLOCK_MONOTONIC's rate by up to several ppm between seconds on the Azure runners, which says nothing about the TSC. The rate over the whole window is measured against the disciplined clock, CLOCK_MONOTONIC on Linux, and must lie within 100 ppm of the rate H3 reports when H3 runs in the same invocation. A Linux host's clocksource is recorded as evidence
H5	Pinned-thread ping-pong rounds over every pair of host CPUs; max_abs_offset_ns is at most 1,000, the measurement is conclusive, and no pair stalls
H6	A microVM with the largest supported vCPU count up to 8 reports a valid NVX-TIME-ABI boot line through nvx-time status and runs nvx-time-probe warp over every CPU pair five times, with all vCPUs halted for 0.1, 1, 5, and 1 s between the runs, so that a host without an invariant TSC corrects the guest TSC as idle host CPUs wake (#265); then a 1-vCPU microVM runs it once. Every run stays within 1,000 ns. --ci-schedule runs CI's schedule instead: two runs 1 s apart
H7	adjtimex reports no STA_UNSYNC and no TIME_ERROR on Linux; w32tm /query /status names a synchronized source on Windows. When H7 can't read either, its detail starts with cannot verify host UTC synchronization: adjtimex fails or reports a clock state Linux doesn't define, or w32tm fails or prints no English Source and Leap Indicator lines, as in a Windows display language other than English

H2, H4, and H5 use a dependency-free host probe, host_time_probe.rs, which the doctor builds with rustc once per source version into $RUNNER_TOOL_CACHE/nvx-host-time-probe (or build/host-time-probe outside CI). H2's profile check, H3, and H6 need the OpenVMM binary, and H3 and H6 also need the guest artifacts; --openvmm, --kernel, and --initrd override their default build paths. --no-openvmm qualifies a host without OpenVMM: H2 then checks the CPU identity and generation only, and H3 and H6 cannot run.

The doctor gates on measured properties, alike on every backend: the guest warp probe at 1 µs with its idle gaps (H6), the TSC rate stability (H4), and the CPU profile (H2 and H3). The host OS's invariant-TSC flags and clocksource are evidence only, in the doctor and in CI. In CI, validate-runner runs the cheap host-level checks H1, H2, and H4 in every job that uses it, on the short --ci-schedule. The microVM and platform jobs run it after downloading OpenVMM and the guest artifacts and add H2's CPU profile check and H3, so that H4 also compares the measured rate with the backend's native rate; the other jobs pass --no-openvmm. The guest warp probe needs a time ABI boot: the platform jobs add H6 on CI's schedule before their benchmarks, and the microVM boot and restore scenarios run CI's warp schedule after every boot and restore and assert its verdict. H5 and H7 remain available for interactive qualification.

Rust toolchain

rust-toolchain.toml pins the exact Rust release that builds OpenVMM, its test guests, the aci_edge_sandboxes and nvx-rs crates, and the host tools that NVX compiles, such as the doctor's time probe and the Windows curl shim. Rustup applies the file to every cargo and rustc run in the checkout, including in the openvmm submodule, and test-openvmm also exports the release as RUSTUP_TOOLCHAIN because flowey can run Cargo outside the checkout. The pin is an exact release rather than stable, so runners provisioned at different times build with the same compiler (#402). Rust 1.99.0 cannot link OpenVMM's guest_test_uefi, failing with undefined symbol: wcslen (rust-lang/rust#163614).

Self-hosted runners keep the release in a trusted RUSTUP_HOME that jobs cannot modify, so the runner setup scripts install it with its OpenVMM targets. validate-runner disables rustup's automatic installation and fails a job before it builds anything when its runner lacks the release, naming the setup script to rerun. The OpenVMM binary cache key includes a hash of the file, so a new release rebuilds OpenVMM, and a change to the file runs the OpenVMM test matrix.

To move the pin, first confirm that guest_test_uefi links with the new release:

rustup toolchain install RELEASE --profile minimal --target x86_64-unknown-uefi
cd openvmm
cargo +RELEASE build -p guest_test_uefi --target x86_64-unknown-uefi --profile release

Then change the release in rust-toolchain.toml, OpenVMMBuildConstants.RUST_TOOLCHAIN, and rust.toolchain in the setup tool manifest, which the setup scripts and the Specula runner setup read, together; a test in scripts/test_nvx_tools.py fails while they differ. Update every self-hosted runner as the setup guide describes before the change merges.

Compilation cache

Every self-hosted runner keeps a persistent sccache cache on its local disk, in _work/_sccache, with the 10-GiB limit that the runner setup scripts configure, and Cargo compiles through it with incremental compilation disabled. CI builds in clean target directories, so each runner compiles a crate only when its own cache lacks it, and loses the cache when its VM is reimaged, as by an image upgrade of its scale set. The OpenVMM jobs that run on these runners start sccache and report its statistics at the end of the job through the sccache action.

sccache keys a Rust compilation on its arguments, its CARGO_* environment, and the contents of its sources and of each library that it names with --extern. It cannot cache proc macros, build scripts, or executables, so every build compiles and links them again. On Linux, the linker reproduces a proc macro's shared library exactly, but MSVC's link.exe stamps every DLL with the time of the link and a random PDB signature. Every Windows build therefore missed the cache for each crate that uses a proc macro directly, several times per job because flowey builds openvmm and pipette in target directories of their own: 243 of the 1,421 cacheable compilations of a WHP VMM-test job, and 286 of the 1,318 of a Windows unit-test job (#693). On Windows, test-openvmm and test-openvmm-unit therefore append /Brepro to _LINK_, which makes link.exe derive both from the image's contents.

Rust crates

The required aci-edge-sandboxes job checks the aci_edge_sandboxes crate on GitHub-hosted Ubuntu and Windows runners through the check-aci-edge-sandboxes action. The action runs rustfmt, Clippy with warnings denied, and rustdoc, all on the Rust release that rust-toolchain.toml pins. It also runs the unit, mock, and fake-OpenVMM integration tests, checks the declared minimum Rust version, and cross-checks the macOS build that MXC compiles. The fake-OpenVMM tests drive the real OpenVMM backend through its control protocol without a hypervisor. It also runs nvx.py test-agent-bundling, which validates edge agent bundle staging with a synthetic bundle, which the bundled-agent tests must find with the digests that were compiled in, and checks that the build rejects a tampered file, a wrong pin, another sandbox ABI, and a link: a symbolic link on Linux and a junction on Windows.

The required nvx-rs job checks the nvx crate, which holds the NVX mechanics that the backend builds on, on the same runners through the check-nvx-rs action. It runs rustfmt, Clippy with warnings denied for all targets and features and for the library without default features, the unit, golden protocol-fixture, and fake-OpenVMM instance tests, rustdoc with warnings denied, the minimum Rust version check, and the macOS cross-check. It also validates bundled artifact staging with a fixture package. Both actions set NVX_BUNDLE=skip for --all-features builds, so that bundled stages nothing instead of downloading the pinned release, and the aci_edge_sandboxes action also sets ACI_EDGE_SANDBOXES_AGENT_BUNDLE=skip, so that agent-bundled does not download an edge agent bundle.

On Linux, the nvx-rs job also lints and tests the guest binaries for x86_64-unknown-linux-musl, the target that guests run, and builds them with the guest profile. The action delegates the musl lint and tests to nvx.py check-guest-binaries, the fresh, path-remapped build, static-ELF check, and digest manifest to nvx.py build-guest-binaries, and the prebuilt-binary integration run to nvx.py test-guest-binaries. The same shared routine supplies every native and Docker guest build, and Alpine, Ubuntu, and Azure Linux ship its nine executables.

The nvx-rs job also checks the runtime adapter in tests/runtime-adapter/, an independent adapter that builds on nvx with a lockfile of its own, default features off, and the public API only. For each of its configurations, protocol-only, host-only, and, on Linux, guest-only for x86_64-unknown-linux-musl, the action runs Clippy with warnings denied, the build, the tests, and a check with the minimum Rust version of nvx, after rustfmt. The tests check that each configuration builds as an independent adapter's does: it enables only its side of nvx, the guest-only build pulls in neither Tokio nor a dependency that only the host side declares, and the guest binary is a static executable. In the guest-only configuration, they also check that the adapter's guest agent serves the control session between the host and the guest with the adapter's own framing, against a test broker that sends it messages that span records, resets, disconnects, and malformed records.

The development release job depends on aci-edge-sandboxes and nvx-rs and requires their combined Linux/Windows results to be successful; failed, cancelled, or skipped crate checks cannot publish a release. Before it uploads anything, it checks that each platform's package ships bin/openvmm and bin/nvx-rs, or their .exe forms on Windows, which download installs.

Each nvx-microvm-tests-{kvm,mshv,whp} job then runs nvx.py test-aci-edge-sandboxes on its self-hosted runner; the debug-kernel jobs skip it, because it boots the production kernel. This command drives a complete provision, start, exec, stop, start, and deprovision cycle of the Alpine guest with the crate's OpenVMM backend. It also checks cancellation, that guest state lasts only until a stop, that a start terminates the VM of an interrupted earlier start, host path mappings, egress rules, and per-execution environments and working directories. On failure it keeps the OpenVMM log under build/test-results/aci-edge-sandboxes-<backend>, which is uploaded with the other microVM logs. Changes under aci_edge_sandboxes/ or nvx-rs/ therefore trigger the NVX microVM test matrix.

The same jobs, except the debug-kernel ones, build the runtime adapter's host-only configuration on their backend and run any ignored tests that it declares, its hypervisor scenarios, with NVX_HYPERVISOR naming the backend. The adapter's other tests, including those of its guest agent, run in the nvx-rs job. Changes under tests/runtime-adapter/ therefore trigger the NVX microVM test matrix too.

Adversarial campaigns

The separate adversarial.yml workflow runs Copilot-driven campaigns only on trusted manual dispatches or schedules from dev. It is not part of pull-request CI. The workflow's dedicated nvx-adversarial-controller runner must already have an authenticated Copilot CLI and an administrator-owned executor wrapper named by the NVX_ADVERSARIAL_EXECUTOR repository variable. The workflow does not install Copilot or initiate login.

The wrapper provisions a distinct disposable KVM, MSHV, or WHP target with no production or GitHub credentials and forwards only the typed executor protocol. Besides the guest and OpenVMM artifacts, the target needs the Rust toolchain and a prebuilt nvx-rs with its provenance for the scenarios that run it, as nvx-rs on the target explains. Existing persistent microVM and performance runners are not valid adversarial targets. Loss of the target heartbeat, a policy oracle, or a teardown/post-campaign boot failure fails the job and requires quarantine and reimage.

Normal Actions artifacts contain only the guest-text-free public summary, catalogued case identifiers, and replay manifest. The external provisioner must collect controller transcripts and complete target logs into access-controlled security storage. See Copilot-driven adversarial testing for the architecture and operational contract.

Copilot environments

copilot-setup-steps.yml prepares the GitHub-hosted runner of Copilot cloud agent sessions, including sessions started from the GitHub web interface. The session's 59-minute limit includes these steps, so they only install pinned tools, restore caches, and stage downloads; they never compile NVX or OpenVMM. The workflow initializes the public OpenVMM submodule without a deploy key, grants access to the runner's /dev/kvm, installs the Python development tools into a virtual environment on the image's Python 3, which validate-nvx also uses, and installs Rust and cargo-nextest at the versions that rust-toolchain.toml, the crate manifest, and the setup tool manifest pin. It restores the shared guest artifacts through the restore-only input of build-guest-artifacts and the KVM OpenVMM binary that build-openvmm caches for the pinned revision, so agents can run microVM tests without rebuilding either. It doesn't build nvx-rs: nvx.py builds it with Cargo, and records its provenance, the first time that a command needs it. The guest artifact restore computes its keys with the same action that saves them. The binary restore repeats the key, paths, and gzip compression of the cache entry that build-openvmm saves, and a test in scripts/test_nvx_tools.py fails when they differ, because a mismatch would otherwise only cause a silent cache miss. Because the agent firewall blocks cdn.kernel.org and cdimage.ubuntu.com, even inside containers, the workflow also installs the native guest build prerequisites, including the pinned Rust toolchain's x86_64-unknown-linux-musl target, allows the unprivileged user namespaces that Alpine's apk uses for package triggers, and stages the pinned Linux and Ubuntu Base archives, so agents can rebuild guest artifacts with build-guest --native. When a runner has a separate /mnt disk with more free space than /, it mounts the workspace there. A failed step would make Copilot skip every later setup step, so each step after checkout continues on error. The last step sets NVX_COPILOT_SETUP to complete or incomplete, lists failed step IDs in NVX_COPILOT_SETUP_FAILED, and fails the run when setup is incomplete. Copilot always runs the version on dev, even for sessions based on other branches, so changes reach agent sessions only after they merge. Pushes run it as a normal workflow for validation when they change the workflow or a file that its steps take versions, requirements, metadata, or code from: the build-guest-artifacts action, all nvx-rs/** files, the files that pin its Rust, cargo-nextest, and shell linter versions, requirements-dev.txt, SOURCE-MANIFEST.json, .gitmodules, the OpenVMM submodule pin, and the NVX CLI and its modules.

Copilot code review uses copilot-code-review.yml instead. Reviews do not build code, so it only checks out the repository and does not install the build and test dependencies used by cloud agent sessions.