feat(checkpoint-restore): add restore_in_place option#282
Open
kamilchodola wants to merge 1 commit into
Open
Conversation
Restore the checkpointed container in place between tests instead of importing the checkpoint archive into a fresh container. The checkpoint is taken locally (Keep, no Export) so the checkpoint data stays in the container's storage directory, and the same container — same name, mounts, and IP — is restored repeatedly, with a zero-timeout stop between tests. This skips the per-test archive extraction and container creation of the export/import path. On a ~7.9G Nethermind checkpoint the per-test podman restore drops from ~15s (of which CRIU page restore was only ~2.9s — the rest was extracting the archive into container storage and creating the container) to roughly the CRIU restore time. Repeated in-place restores are safe: podman accepts a restore of a stopped container whenever its kept checkpoint data is present, restores resume from the identical memory state each time, and the container IP is stable (verified on podman 5.x + CRIU 4.x: checkpoint, restore --keep, kill, restore --keep, ... loops indefinitely). tmpfs_threshold / tmpfs_max_size only apply to the export archive and are ignored in this mode. Default remains the export/import path (restore_in_place: false).
kamilchodola
added a commit
to NethermindEth/gas-benchmarks
that referenced
this pull request
Jul 14, 2026
Stateful suites now get checkpoint-restore with restore_in_place out of the box (rollback auto already maps compute to 'none', so compute runs carry no checkpoint burden). Upstream benchmarkoor binaries ignore the option until ethpandaops/benchmarkoor#282 merges and simply use the export/import path; the fork ref enables the ~4s in-place restores now.
kamilchodola
added a commit
to NethermindEth/gas-benchmarks
that referenced
this pull request
Jul 15, 2026
… per-test summary, dotTrace profiling) (#159) * feat: add Benchmarkoor - Nethermind workflow (podman + CRIU memory checkpoints) Integrates ethpandaops/benchmarkoor for Nethermind only, running on the stateful-generator self-hosted runner. - container_runtime: podman with rollback_strategy: container-checkpoint-restore by default for stateful suites: pre-run steps (gas-bump/funding) are baked into a CRIU memory checkpoint + ZFS datadir snapshot, then every test gets an instant rollback/restore instead of a full container recreate. - Bootstraps a file-backed zpool on the runner and seeds a dataset from /mnt/sda/<network>/nethermind when the snapshot is not already on ZFS (one-time rsync, reused across runs). - Tests/opcodes pulled from gas-benchmarks release archives (default amsterdam-repricings-v5.2.0, jochemnet), chainspec from scripts/genesisfiles/nethermind at the triggering commit. - Rollback strategy selectable (checkpoint-restore / container-recreate / rpc-debug-setHead / none) for A/B comparison; extra_run_config input deep-merges arbitrary benchmarkoor YAML last. - Results uploaded as workflow artifact + job summary via the ethpandaops/benchmarkoor composite action, which also installs podman/CRIU/crun and starts the rootful podman socket. * fix: auto-size and grow the file-backed zpool, resume interrupted seeds The jochemnet nethermind snapshot is ~900G; the fixed 500G pool filled at 57% of the seed copy. Pool size now defaults to 'auto' (snapshot + 15% + 32G) with a free-space preflight, existing pools are grown in place (sparse truncate + zpool online -e), and seed completion is tracked via the gb:seeded ZFS user property so interrupted copies resume instead of being mistaken for complete. Also add --Init.BaseDbPath=/data to the default Nethermind flags: the runner snapshots hold the nethermind_db content at their root, which the stock --datadir=/data would miss. * fix: Nethermind BaseDbPath must include the network subdir (/data/mainnet) Nethermind's BaseDbPath convention already contains the per-network dir (mainnet config uses nethermind_db/mainnet; our own scripts/nethermind/docker-compose.yaml uses /nethermind/data/mainnet). With --Init.BaseDbPath=/data the client created an empty DB next to the snapshot's mainnet/ dir and booted at genesis (chain head 0, endless SYNCING). Point it at /data/mainnet instead. * fix: take corrected jochemnet chainspec from fix/hook-block-testing-only-scenarios main's copy schedules the Amsterdam EIP set (missing 7954/7976/7981/8037) at 0x697D1108, which falls inside the snapshot's synthetic block range — the node then computes BAL hashes for stored blocks that have none and rejects them (InvalidBlockLevelAccessListHash). The corrected file activates 7708,7778,7843,7928,7954,7976,7981,8024,8037 at 0x697ddeff (head block 24402727 timestamp + 1), matching the chainspec ethpandaops use for this exact snapshot (verified params/genesis/accounts-identical to skylenet's devnet-6 gist). * chore: stop mirroring client logs into the CI job log Full Nethermind output is always written to runs/<run_id>/container.log in the results artifact; client_logs_to_stdout only duplicates it into the workflow log, which drowns the job output during the startup block replay. Re-enable per run via extra_run_config when live-debugging. * feat: allow building benchmarkoor from a custom git repo Needed to validate benchmarkoor branches (e.g. the restore_in_place fork) on the runner before they land upstream. * fix: queue new dispatches instead of cancelling in-progress benchmarks cancel-in-progress: true killed a 3h validation run when the next dispatch arrived on the same ref. Benchmarks should queue. * fix: force source build when benchmarkoor_git_ref is set GitHub substitutes the default for explicitly-empty dispatch inputs, so passing benchmarkoor_image='' still pulled ghcr master and silently skipped the fork build. A non-empty git ref now implies image=''. * feat: make restore_in_place the built-in default for stateful runs Stateful suites now get checkpoint-restore with restore_in_place out of the box (rollback auto already maps compute to 'none', so compute runs carry no checkpoint burden). Upstream benchmarkoor binaries ignore the option until ethpandaops/benchmarkoor#282 merges and simply use the export/import path; the fork ref enables the ~4s in-place restores now. * feat: per-test MGas/s job summary, dotTrace + trace_blocks profiling - Append a per-test throughput table (median/min/max + collapsible all-tests table) to the GitHub job summary, generated from the run's result.json by .github/scripts/benchmarkoor-pertest-summary.jq. - diagnostics_mode input (dottrace/dotmemory/dotnet-trace) runs the client through the diag-image DIAG_WITH entrypoint; snapshots are collected via the fork's new instance-level extra_mounts option (bind mount at /nethermind/diag), uploaded as an artifact, and dotTrace snapshots are converted to XML on windows-latest with Reporter.exe (mirrors nethermind's run-expb-reproducible-benchmarks). - trace_blocks input (implies dottrace) injects NETHERMIND_PROFILE_BLOCKS for the BlockProfiler plugin (NethermindEth/nethermind#12444): per-block dotTrace windows so only testing blocks are traced, excluding gas-bump and setup blocks; 'auto' maps to the jochemnet testing blocks 24407730,24407731. - Profiling runs resolve rollback auto -> rpc-debug-setHead (one live process; dotTrace is incompatible with CRIU restores) and default benchmarkoor to the fork's gas-benchmarks branch for extra_mounts. * fix: match chunked dotTrace snapshot index in XML conversion The diag entrypoint's --save-to base produces a chunked snapshot: an extensionless 'dottrace' index plus dottrace.XXXX chunk files. Match the index (plus *.dtp/*.dtc) like repricing-nethermind does instead of requiring a .dtp extension. * fix: match Reporter version to the diag image's dotTrace CLI masterdiag profiles with dotTrace 2026.1.4; Reporter 2026.1.0.1 rejects its snapshots ('unknown or unsupported snapshot format'). Also make the XML step's exit explicit: fail only when snapshots existed but nothing converted, instead of leaking the last Reporter exit code. * fix: bypass masterdiag's timeline profiling for dottrace XML conversion masterdiag's diag-entrypoint profiles with --profiling-type=timeline; Reporter.exe cannot read Timeline snapshots (MFDTEM storage, 'Invalid storage type'), verified by local bisection: default/sampling capture produces MFDTPF storage that converts fine. In dottrace mode the container entrypoint is now overridden with an expb-style wrapper: default (sampling) profiling, .dtp snapshot naming, and --use-api only when NETHERMIND_PROFILE_BLOCKS is set (MeasureProfiler per-block windows need API mode; API mode suppresses whole-run collection, so trace_blocks needs an image with the BlockProfiler plugin). * fix: flat-array dottrace entrypoint — benchmarkoor expands $vars in config benchmarkoor os.Expand()s the raw config text at load time, so the bash wrapper's runtime variables ($tool, $c, $@) were blanked before the container ever started ('dottrace tool not found in image'). Replace the shell wrapper with a flat argv entrypoint — the client flags (benchmarkoor's command) are appended by the runtime, ENTRYPOINT+CMD style — and decide --use-api at compose time. The dottrace path (/opt/diag-tools/dottrace) is where nethermind diag images install it (verified in masterdiag and bal-devnet-7-diag). * docs: diagnostics-run numbers are not comparable to checkpoint-restore runs rpc-debug-setHead keeps client caches warm across tests, so only the first test in a profiling run measures cold state (observed ~4x speedup on subsequent tests). Single-test filters give pure-cold profiles.
skylenet
self-requested a review
July 16, 2026 14:07
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(checkpoint-restore): add
restore_in_placeoptionWhat
Adds an opt-in
checkpoint_restore_strategy_options.restore_in_placeflag to thecontainer-checkpoint-restorerollback strategy. When enabled, the checkpoint is taken locally (Keep, noExport) so the CRIU images stay in the container's storage directory, and the same container — same name, mounts, and IP — is restored in place before every test, with a zero-timeout stop between tests. Default (false) keeps the current export/import behavior unchanged.Why
On the export/import path, every test pays for extracting the checkpoint archive into container storage plus creating and wiring a fresh container — CRIU itself is a small fraction of the per-test restore:
Measured on a Nethermind instance with a ~7.5 GB memory image (1.89 M pages) over the jochemnet stateful suite (gas-benchmarks repricing tests), identical 11-test filter, same host,
tmpfs_thresholdactive on the export/import side. For a 550-test suite this saves ~100 minutes of pure restore overhead per run.Podman semantics (verified empirically)
The design relies on podman restoring a stopped container repeatedly from kept checkpoint data. Verified on podman 5.x + CRIU (ubuntu-24.04 and a self-hosted runner):
podman container checkpoint <ctr>→ (podman container restore --keep <ctr>→podman kill/podman stop -t 0)×N loops indefinitely; the process resumes from the identical memory state every time (counter-in-a-loop probe), theCheckpointed=falsestate flag after the first restore does not block subsequent restores, and the container IP is stable across restores.Notes:
--nameis not passed on the in-place restore (import-only option).tmpfs_threshold/tmpfs_max_sizeonly concern the export archive and are documented as not applying in this mode.restart_containercomposes with this option as before.Changes
pkg/podman:CheckpointContainerLocal+RestoreContainerInPlace(with CRIU stats logging, mirroring the existing methods)pkg/runner/strategy_checkpoint.go: branch the checkpoint phase, per-test restore, and per-test teardown (stop instead of remove)pkg/config: option field +GetCheckpointRestoreInPlacegetter + unit testsdocs/configuration.md,config.example.yaml: documentationui:restore_in_placeinCheckpointRestoreStrategyOptionstype and the RunConfiguration panel