WIP: add native rust-based initramfs generator, init and qemu wrapper#258
Open
ddiss wants to merge 289 commits intorapido-linux:masterfrom
Open
WIP: add native rust-based initramfs generator, init and qemu wrapper#258ddiss wants to merge 289 commits intorapido-linux:masterfrom
ddiss wants to merge 289 commits intorapido-linux:masterfrom
Conversation
This significantly simplifies archiving. Any hardlink will be treated as a normal file: assigned a unique inode number and associated data segment. Data segments may be duplicated if multiple hardlinks with the same source inode are archived (easy to avoid). The main benefit here is that deferred I/O for hardlink data segments is removed, so src-path = archive-path behaviour can finally be dropped. Signed-off-by: David Disseldorp <ddiss@suse.de>
Following removal of dracut-cpio hardlink support, remove GNU cpio vs dracut-cpio output comparison tests. Leave a single remaining hardlink tests, which simply checks that the extracted output data remains the same. Signed-off-by: David Disseldorp <ddiss@suse.de>
This is the final step in removing src-path-is-dest-path archiving behaviour for the core cpio library; the archive_symlink caller is responsible for providing the symlink target path. Signed-off-by: David Disseldorp <ddiss@suse.de>
Make state.off private. Also, use consts for the trailer name string. Signed-off-by: David Disseldorp <ddiss@suse.de>
It's only used by dracut-cpio based on a command line parameter, so change it to a local variable. Signed-off-by: David Disseldorp <ddiss@suse.de>
Move list_separator out of the core cpio library and into dracut-cpio. Don't allow it to be changed at runtime, as doing so would mean reworking too many callers. Drop the one test which checks for nul-separators. Signed-off-by: David Disseldorp <ddiss@suse.de>
Passing both of these around everywhere doesn't make much sense, as the properties should be static. Signed-off-by: David Disseldorp <ddiss@suse.de>
We can pass in an &str. Signed-off-by: David Disseldorp <ddiss@suse.de>
I'm finally getting my head around the String->PathBuf &str->&Path mappings. Signed-off-by: David Disseldorp <ddiss@suse.de>
Add parent directories to initramfs as they are encountered for files. Use a new paths_seen map to filter duplicates. Signed-off-by: David Disseldorp <ddiss@suse.de>
Parameter parsing only for now. Signed-off-by: David Disseldorp <ddiss@suse.de>
The fs::Metadata type is private, which makes it difficult to create cpio archive entries for items which don't exist locally. Create a new public ArchiveMd type for this purpose, which can easily be instantiated from fs::Metadata . This also has the benefit of moving some of the duplicate 64-bit -> 32-bit (cpio) overflow checks into a single function. There's no backward compatibility offered, so this change will break existing callers. Signed-off-by: David Disseldorp <ddiss@suse.de>
Signed-off-by: David Disseldorp <ddiss@suse.de>
Signed-off-by: David Disseldorp <ddiss@suse.de>
We now have the ability to mock up ArchiveMd metadata for cpio entries, so src→dest install can be properly implemented. Support is still a bit ugly in that symlinks are not handled - it gets a bit messy / complex if we need to modify target paths, so I'd prefer to skip it if we can get away with it. Signed-off-by: David Disseldorp <ddiss@suse.de>
Without a workspace, Cargo appears to only supports a single lib. cpio is selfcontained, so just split it out as a separate (internal) crate. Signed-off-by: David Disseldorp <ddiss@suse.de>
Implement a rust-style iterator API which walks through each entry in the Reader-provided cpio archive. For simplicity, only metadata (including filename) is available to the caller. Rapido VM CPU, network and memory resources are specified in images via special file paths, so data segments can be ignored for now. Signed-off-by: David Disseldorp <ddiss@suse.de>
namesize must include a zero terminator. Not sure how other archivers behave here. Signed-off-by: David Disseldorp <ddiss@suse.de>
GNU cpio adds zeros after the trailer, which causes a header parsing failure if we continue. Signed-off-by: David Disseldorp <ddiss@suse.de>
Avoid calling GNU cpio --list and instead use the new cpio iter API. Further optimizations can be made to stop traversing after all rapido-rsc entries have been covered. Signed-off-by: David Disseldorp <ddiss@suse.de>
Signed-off-by: David Disseldorp <ddiss@suse.de>
Add some simple tests for it too. Signed-off-by: David Disseldorp <ddiss@suse.de>
Stripping '/' path prefixes makes sense for initramfs where we extract from root. Signed-off-by: David Disseldorp <ddiss@suse.de>
Tests which use absolute paths and compare binary output to GNU cpio need to be reworked, such that GNU cpio also omits the '/' prefix. This is done by changing to the root directory. When cpio.rs test coverage improves, we can probably drop dracut-cpio. But cpio.rs tests have a long way to go before we're at that point. Signed-off-by: David Disseldorp <ddiss@suse.de>
When determining VM cpu, memory and network resources, we don't need to traverse the entire cpio archive if we know that all rapido-rsc entries are located together. Assume this is the case (it is with Dracut) and break traversal when stepping past a previously encountered rapido-rsc path. This might be a premature optimization, in which case I'm sorry. Signed-off-by: David Disseldorp <ddiss@suse.de>
Signed-off-by: David Disseldorp <ddiss@suse.de>
rapido-init doesn't support systemd yet. It *does* support networking with systemd-networkd though, so should only break a few test cases. Signed-off-by: David Disseldorp <ddiss@suse.de>
Signed-off-by: David Disseldorp <ddiss@suse.de>
We'll likely need to split this helper out for others, so add some basic test coverage. Signed-off-by: David Disseldorp <ddiss@suse.de>
It now returns a bool which indicates whether it processed a rapido-rsc path. Check it. Signed-off-by: David Disseldorp <ddiss@suse.de>
ddiss
added a commit
to ddiss/linux
that referenced
this pull request
Feb 26, 2026
Github hosted "ubuntu-latest" x86-64 runners have enough resources (4 core, 16G RAM, 14 GB SSD) to build and run mainline kernel + xfstests in a nested VM. This script uses rapido rapido-linux/rapido#258 as a minimal initramfs generator and thin wrapper around QEMU. There are some test failures due to the minimal test environment (e.g. no udev yet). TEST and SCRATCH devices are backed by 8G zstd compressed qcow2 images. Signed-off-by: David Disseldorp <ddiss@suse.de>
This omits the old dracut-based fstests runners for now, so failure to start udevd isn't fatal. Signed-off-by: David Disseldorp <ddiss@suse.de>
Needed for generic/478. Signed-off-by: David Disseldorp <ddiss@suse.de>
Dracut no longer does this for us. Signed-off-by: David Disseldorp <ddiss@suse.de>
Needed for generic/492 and generic/598 respectively. Signed-off-by: David Disseldorp <ddiss@suse.de>
Signed-off-by: David Disseldorp <ddiss@suse.de>
This allows us to run a bunch of tests which are otherwise skipped via _require_mknod(). Signed-off-by: David Disseldorp <ddiss@suse.de>
We might be able to avoid the open() by using something like:
enum BufIn {
File(io::BufReader<fs::File>),
Stdin(io::BufReader<io::Stdin>),
}
struct ArgsState {
manifests: Vec<BufIn>,
I don't think it's worth the extra complexity though.
Signed-off-by: David Disseldorp <ddiss@suse.de>
This isn't a public API, so there's no need to worry. Signed-off-by: David Disseldorp <ddiss@suse.de>
In preparation for supporting explicit path filters, we should ensure that pretty much everything gets put in paths_seen. I'd be interested in measuring the perf / resource effects of this change, but don't have time atm. Signed-off-by: David Disseldorp <ddiss@suse.de>
filter can be used to stop path traversal at specific points within a tree. This is particularly useful for source directories with .git/ artifacts. Link: rapido-linux#256 Signed-off-by: David Disseldorp <ddiss@suse.de>
Use the new "filter" directive to avoid archiving fstests git artifacts during $FSTESTS_SRC traversal. This reduces the initramfs size considerably. I.e. before 161835528, after 134994440 = ~25M saved. Link: rapido-linux#256 Signed-off-by: David Disseldorp <ddiss@suse.de>
This is helpful for rapido-rsc paths. Signed-off-by: David Disseldorp <ddiss@suse.de>
The "file" manifest directive no longer requires a local source for archiving, so drop the silly "dracut.conf.d/.empty" uses. Signed-off-by: David Disseldorp <ddiss@suse.de>
A few currently-skipped tests need these to run. flock is part of util-linux so can be listed as mandatory. Signed-off-by: David Disseldorp <ddiss@suse.de>
ddiss
added a commit
to ddiss/linux
that referenced
this pull request
Feb 27, 2026
Github hosted "ubuntu-latest" x86-64 runners have enough resources (KVM, 4 cores, 16G RAM, 14 GB SSD) to build and run mainline kernel + xfstests in a nested VM. This script uses rapido rapido-linux/rapido#258 as a minimal initramfs generator and thin wrapper around QEMU. For simplicity it'd likely make sense to branch it under the btrfs namespace. The test VM currently uses btrfs-progs from the ubuntu-24.04 host system. This could also be changed to a source-compiled version. TEST and SCRATCH devices are backed by 8G zstd compressed qcow2 images. Signed-off-by: David Disseldorp <ddiss@suse.de>
Collaborator
Author
|
I think these changes are pretty much ready for inclusion now, with systemd init support the only blocker. Existing dracut scripts will continue to work here, but use rapido-init and rapido-vm (as qemu wrapper) for boot. I'll branch before merge so that people can continue to use the unmodified Dracut boot path. |
btrfs/058 calls xfs_io interactively. If the termcap file is missing then libedit prints an error: Cannot read termcap database; using dumb terminal settings. Add the linux termcap file, corresponding to rapido-init's TERM env. It may make sense to use "ncurses6-config --terminfo-dirs" based paths, but this should be fine for now. Signed-off-by: David Disseldorp <ddiss@suse.de>
This allows us to run a few more skipped tests. unshare is from util-linux, so make it a hard dependency. Use hyphens instead of underscores in dm kmod names to match actual module name. Signed-off-by: David Disseldorp <ddiss@suse.de>
Relative symlink target paths are resolved relative to the symlink path. At the moment, any relative "slink" target path will be processed as an absolute path relative to the host working directory. Fix this. Signed-off-by: David Disseldorp <ddiss@suse.de>
I don't think we require these, and the /run mount means that we can't pack pre-created /run paths such as /run/systemd in the initramfs image without them disappearing. Signed-off-by: David Disseldorp <ddiss@suse.de>
We no longer mount over /run, so use the "dir" manifest directive to ensure that these paths exist on boot. Signed-off-by: David Disseldorp <ddiss@suse.de>
This better reflects distro defaults. Symlinks are left as 0o777. Signed-off-by: David Disseldorp <ddiss@suse.de>
systemd-udevd --daemon now fails to start, due to a lack of /run: Failed to create /run/udev: No such file or directory I don't think adding it back to rapido-init makes sense, so add it to the fstests-btrfs cpio manifest. Fixes: 55f0cae ("rapido-init: drop /dev/shm and /run default tmpfs mounts") Signed-off-by: David Disseldorp <ddiss@suse.de>
If udevd fails to come up then xfstests _udev_wait() will wait forever. Signed-off-by: David Disseldorp <ddiss@suse.de>
Aside from systemd-networkd, systemd (proper) appears to need it to avoid the "first-boot" boot path. Signed-off-by: David Disseldorp <ddiss@suse.de>
rapido-vm.rs replaces it, so we can get rid of the original QEMU wrapper script. Signed-off-by: David Disseldorp <ddiss@suse.de>
rapido-init is now used across all rapido-cut and dracut based images, so we can drop the old pre-autorun init logic. 00-rapido-init.sh, which used to be the dracut-hook entry point can also be dropped. What remains in vm_autorun.env are mostly just network helper functions called from some autorun scripts, which could be moved out into autorun/lib if necessary. Signed-off-by: David Disseldorp <ddiss@suse.de>
This is a regression fix for dracut based images, most likely introduced when we switched from the dracut init hook to rapido-init. systemd-networkd needs a config under /etc/systemd/network and writes state to /run/systemd/netif , so ensure that the parent directories exist in Dracut created images. manifest/net.fest already handles this for rapido-cut created images. Signed-off-by: David Disseldorp <ddiss@suse.de>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sangeetha and I have been working on some changes to replace Dracut with
a rust-based initramfs / cpio image generator. The main reasons for this
are:
locally available
modules, and kernel / user dependency gathering for cpio
and is mostly written in bash
The rewrite builds on my previous dracut-cpio implementation and adds:
parsing (thanks @thackara !)
scripts
It's otherwise kept as minimal as possible, with rust-elf and the std
library the only major external dependencies. One single-file crosvm
argument parser is also bundled.
The preliminary benchmark results look good, particularly for initramfs
image generation (cut):
To try these changes yourself, check out this branch and run:
You may need to change your rapido.conf file a little if you
use env variables or shell callouts.
I'm flagging this as WIP, as there are still a few things to do:
in the working directory
I don't expect to convert remaining cut scripts before merge. Dracut and
rust based functionality should be able to live side by side, although
rapido.conf parsing is much less flexible in rust: no invocations,
currently no env var expansion, variables must be wrapped in {}.