install to-disk: final umount fails with EBUSY because bootc still holds mmap'd ostree .dirtree objects on the target filesystem

[jonassvatos]

Description

bootc install to-disk fails at the very end, in the Unmounting filesystems task:

Trimming root
.: 10 GiB (10785652736 bytes) trimmed
Finalizing filesystem root
Unmounting filesystems
umount: /run/bootc/mounts/rootfs: target is busy.
error: Installing to disk: Task Unmounting filesystems failed: ExitStatus(unix_wait_status(8192))

The cause is that the bootc process itself still holds memory-mapped ostree metadata objects that live on the target filesystem when it executes umount -R. An mmap pins the mount without any visible open file descriptor, so the unmount fails with EBUSY — deterministically, not as a race.

Diagnosis

I bind-mounted a diagnostic wrapper over /usr/bin/umount inside the install container. The wrapper passes through every call except the final umount -R, for which it scans for holders (fd/cwd/root/exe links, /proc/*/maps, mountinfo of other namespaces, loop devices, swap) and retries. At the moment of failure:

• No process anywhere had an open fd, cwd, root, or exe on the target mount.
• No loop device or swapfile was backed by it; no other mount namespace held a clone.
• The parent bootc process (the one that spawned umount) had these mappings in /proc/<pid>/maps:

7febb5b35000-7febb5b3a000 r--p 00000000 fc:13 19705981  /run/bootc/mounts/rootfs/ostree/repo/objects/a5/f65aded760a88e3958c90cad66f9aa3eec51c6c3d143c6a01b1f8f3083548b.dirtree
7febb5b3a000-7febb5b41000 r--p 00000000 fc:13 19694781  /run/bootc/mounts/rootfs/ostree/repo/objects/87/c38c9e136d34f5a2c444c9d49c2788ee424fa5adaa7e89c57b3f73d3c834e7.dirtree
7febb5b41000-7febb5b46000 r--p 00000000 fc:13 19705981  /run/bootc/mounts/rootfs/ostree/repo/objects/a5/f65aded760a88e3958c90cad66f9aa3eec51c6c3d143c6a01b1f8f3083548b.dirtree
7febb5b46000-7febb5b4d000 r--p 00000000 fc:13 19694781  /run/bootc/mounts/rootfs/ostree/repo/objects/87/c38c9e136d34f5a2c444c9d49c2788ee424fa5adaa7e89c57b3f73d3c834e7.dirtree

Two .dirtree objects from the target repo, each mapped twice — presumably GVariant maps created by libostree during the deploy and still referenced (cached) at unmount time. They stayed mapped across three retries spanning ~30 seconds (always EBUSY), i.e. this is not a transient writeback race.

umount -l succeeded immediately afterwards and the installation then completed; since finalize_filesystem has already remounted the filesystem read-only and run fsfreeze -f/-u at that point, a lazy unmount appears safe there.

Why only some images hit it

The same flow with quay.io/almalinuxorg/atomic-desktop-gnome:10 (bootc 1.15.2) succeeds, but a derived image (same base, ~260 additional RPMs, 85 layers / 2.5 GB compressed) fails every time, with identical bootc inside. The leaked .dirtree objects are ~20–28 KB; the apparent trigger is simply that the bigger image has directory-tree metadata objects large enough to take (and keep) libostree's mmap path, while the base image's stay small. So the bug surfaces content-dependently, which makes it look like a problem with the user's image when it isn't.

Reproduction

Any sufficiently large bootc image appears to work. Mine is private, but the recipe is: FROM quay.io/almalinuxorg/atomic-desktop-gnome:10, then one dnf install layer adding ~260 packages (EPEL + RPM Fusion: ffmpeg, vlc, firefox, gcc, ImageMagick, @multimedia, freeipa-client, …), then:

bcvk to-disk --format qcow2 <image> out.qcow2

(bcvk 0.10.0 — but bcvk is just the vehicle; the failing umount is inside bootc install to-disk, run by podman in the ephemeral VM. Reproduced with a raw target disk as well, so it is not qcow2-related.)

Suggested fix

Drop/unref whatever still holds the mapped target-repo variants (sysroot/repo references) before the Unmounting filesystems task in install_to_disk() — the existing comment there ("Drop all data about the root except the bits we need to ensure any file descriptors etc. are closed") suggests that was the intent, but mmaps survive it. Alternatively (or additionally), fall back to umount -l after finalize_filesystem() has remounted read-only and flushed the journal, since the mappings are read-only and the process exits right after.

I have the full instrumented log (ancestor fd tables, holder scans for each retry, mountinfo dumps) and am happy to attach it or test patches.

System details

• bootc 1.15.2 (inside both images); no related fix found in v1.16.0
• Host: AlmaLinux 10, kernel 6.12.0-211.7.4.el10_2.x86_64, podman + bcvk 0.10.0
• Target filesystem: xfs (default layout from bootc install to-disk --generic-image)

[jonassvatos]

Root cause found — this is actually a libostree bug, not bootc itself.

The opaque-whiteout scan in checkout_tree_at_recurse() (ostree-repo-checkout.c) breaks out of a g_variant_iter_loop() without freeing the current element, leaking a reference to the extracted checksum variant for every directory containing an opaque whiteout entry (.wh..wh..opq). The leaked child variant pins the parent dirtree object's backing GMappedFile, and since the scan only runs with process_whiteouts enabled, it only fires when checking out derived container layers — once for the merged tree and once for layer relabeling, which is why each object was mapped exactly twice and why pure base images (no derived layers) install fine.

In my image the dnf layer carries opaque whiteouts in /usr/include/linux and /usr/lib/.build-id, whose dirtrees exceed the ~16kB mmap threshold — those two mappings kept the target filesystem busy at the final umount -R.

Fix submitted upstream: ostreedev/ostree#3595. Verified both with an instrumented ostree-ext container image pull (zero residual dirtree mappings at every import phase) and by re-running the previously 100%-reproducible bootc install to-disk with the patched library bind-mounted — it completes cleanly with no other change.

Possibly still worth considering on the bootc side: a retry-then-umount -l fallback in the install finalization for robustness against any future fd/mapping leak, since at that point the filesystem is already remounted read-only and fsfreeze'd.