Skip to content

feat: enable ABUpdate for ACL#28

Open
bfjelds wants to merge 25 commits into
aclmainfrom
user/bfjelds/ab-verity-partuuid-uki-addons-20260708
Open

feat: enable ABUpdate for ACL#28
bfjelds wants to merge 25 commits into
aclmainfrom
user/bfjelds/ab-verity-partuuid-uki-addons-20260708

Conversation

@bfjelds

@bfjelds bfjelds commented Jul 9, 2026

Copy link
Copy Markdown
Member

Summary

Enables Trident-driven A/B updates on ACL images by giving /usr dm-verity a slot-agnostic boot path, installing/enabling Trident in the image, and expanding the ESP to support A/B UKI staging on arm64.

Changes

Verity: dedicated hash partitions + slot-agnostic boot

  • disk_layout_uki.json: add dedicated verity hash partitions HASH-A and HASH-B (type dps-usr-verity) paired to USR-A/USR-B via a new verity_hash cross-reference; renumber OEM/ROOT accordingly.
  • disk_util: add dps-usr-verity partition-type GUID resolution and the verity_hash layout key; add a separate-hash-partition verity mode (hash tree written to the dedicated partition instead of an inline hash-offset); emit the btrfs filesystem UUID and verity superblock UUID via new --fs_uuid/--verity_uuid flags; only the active (prioritized) slot is formatted at build time.
  • build_image_util.sh: capture the new UUID outputs and pass them to bootloader install.

UKI / systemd-boot (the boot path for ACL images). rpm/uki_install.sh:

  • The main UKI cmdline carries only slot-independent args (mount.usr=/dev/mapper/usr, root=LABEL=ROOT, ...).
  • Slot-specific verity identity is delivered via a systemd-stub addon (verity.addon.efi in <uki>.efi.extra.d/) built per slot with systemd.verity_usr_data=PARTUUID=<usr> and systemd.verity_usr_hash=PARTUUID=<hash> plus the slot root hash. Slot A is active by default; Trident switches slots by swapping the active addon. This lets one UKI boot either slot without re-signing.

GRUB (secondary bootloader path). grub_install.sh (declare --fs_uuid / --verity_uuid), rpm/grub_install.sh (read and inject the values), and rpm/grub.cfg (template @@FSUUID@@ / @@VERITYUUID@@): build the verity cmdline from systemd.verity_usr_data=UUID=<fs> and systemd.verity_usr_hash=UUID=<verity> instead of a hard-coded PARTUUID + hash-offset.

ESP capacity for A/B UKI staging

  • disk_layout_uki.json: increase the EFI System Partition from 128 MiB (262144 blocks) to 256 MiB (524288 blocks).
  • The arm64 UKI is approximately 82 MiB, compared with approximately 49 MiB on amd64. During an A/B update, the new UKI is staged on the ESP alongside the running UKI; the 128 MiB ESP runs out of space on arm64. The 256 MiB ESP accommodates two arm64 UKIs plus bootloader files and headroom, while remaining 4096-block aligned.

Trident enablement

  • rpm/package_catalog.yaml: add the trident package.
  • rpm/build_image_util.sh: enable tridentd.socket.

arm64 test-image boot reliability

  • vm_image_util.sh: add install_uki_timeout_addon, a UKI addon that raises systemd.default_device_timeout_sec=120. Scoped to the arm64 kola test image only; it fixes a CI-only QEMU-TCG emulation flake where udev cannot initialise the ESP/OEM/usr-verity devices within the default device timeout and the VM drops to an emergency shell. Production and amd64 boot behaviour are unchanged.

Docs

  • acl/docs/architecture.md: document the dedicated hash partition and slot-agnostic verity boot.

Required PRs

Validation

bfjelds and others added 4 commits July 8, 2026 21:40
…integration

Move dm-verity from inline hash mode (appended to USR partition) to
dedicated HASH-A/HASH-B partitions. Switch boot cmdline from partition
UUIDs (PARTUUID=) to filesystem/verity UUIDs (UUID=) so the same UKI
works for both A and B slots, enabling A/B updates.

Changes:
- disk_layout_uki.json: Add HASH-A (3) and HASH-B (5) partitions with
  dps-usr-verity type, renumber OEM→6 and ROOT→7
- disk_util: Add DPS_USR_VERITY_GUIDS, support verity_hash cross-ref
  to dedicated hash partition, extract verity/fs UUIDs
- build_image_util.sh: Pass --verity_uuid and --fs_uuid through the
  verity and bootloader pipeline
- grub.cfg: Use UUID= instead of PARTUUID= with hash-offset
- grub_install.sh: Inject @@FSUUID@@ and @@VERITYUUID@@ placeholders
- uki_install.sh: Read UUIDs from build artifacts instead of static
  disk_layout.json values, add --verity_uuid/--fs_uuid flags
- package_catalog.yaml: Add trident to systemd package group
- build_image_util.sh (rpm): Enable tridentd.socket at build time
- rpm_install.sh: Temporarily disable SELinux module removal to
  unblock trident install

Based on changes from user/frhuelsz/ab-update-trident branch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move systemd.verity_usr_data, systemd.verity_usr_hash, and usrhash
from the main UKI cmdline into per-slot UKI addons. This enables
trident to switch A/B slots by replacing the active addon.

The main UKI cmdline now contains only slot-independent args:
  mount.usrflags=ro root=LABEL=ROOT rootflags=rw consoleblank=0

Each verity addon contains the slot-specific args:
  mount.usr=/dev/mapper/usr
  systemd.verity_usr_data=PARTUUID=<data-partition>
  systemd.verity_usr_hash=PARTUUID=<hash-partition>
  systemd.verity_usr_options=panic-on-corruption
  usrhash=<root-hash>

Build artifacts:
  EFI/Linux/<uki>.extra.d/verity.addon.efi  (active, slot A)
  acl/uki-addons/verity-a.addon.efi         (template)
  acl/uki-addons/verity-b.addon.efi         (template)

Partition UUIDs are looked up by label (USR-A, HASH-A, etc.) from
disk_layout_uki.json for robustness against layout renumbering.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…verity

Address deep review findings DR-001, DR-002, DR-003, DR-006:

- DR-002: Restore die guards for FLAGS_fs_uuid and FLAGS_verity_uuid.
  grub_install.sh only warns on missing UUIDs and silently substitutes
  empty values into grub.cfg.  These guards are the only fail-fast gate.

- DR-001: Move mount.usr=/dev/mapper/usr from addon cmdline to main UKI.
  The mapper device name is slot-independent; only the PARTUUID and root
  hash are slot-specific.

- DR-003: Collapse dead if/else that produced identical cmdline in both
  branches (vestigial from pre-addon refactoring).

- DR-006: Replace silent non-verity else branch with die.  ACL UKI is
  always verity-enabled; a non-verity path would produce a broken image.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Address deep review findings DR-004, DR-005, DR-007, DR-008:

- DR-004/005: Replace bare jq lookups with _lookup_partuuid() helper that
  validates JSON structure (.layouts.base exists), rejects duplicate labels,
  and uses jq -e to catch null/missing uuid fields.

- DR-007: Move mktemp -d after all validation die paths so early failures
  don't leak temp directories.

- DR-008: Add comment clarifying that both slot templates share the same
  root hash at build time and trident overwrites at update time.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@bfjelds
bfjelds changed the base branch from user/bfjelds/ab-trident-verity-hash-partitions-20260708 to aclmain July 13, 2026 20:38
@bfjelds bfjelds changed the title feat: move verity kernel command line params into A/B PARTUUID-based UKI addons feat: enable ABUpdate for ACL Jul 13, 2026
Uncomment the `semodule -X 100 -r` block in rpm_configure_selinux that
was temporarily disabled to unblock the Trident install, restoring
removal of unnecessary SELinux policy modules.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR enables Trident-driven A/B updates for Azure Container Linux (ACL) images by making /usr dm-verity boot slot-agnostic (primary UKI path via systemd-stub addons) and by updating the GRUB fallback cmdline construction to avoid hard-coding partition identity/inline hash offsets. It also adds Trident to the image and enables its socket.

Changes:

  • Introduce dedicated /usr dm-verity hash partitions (HASH-A / HASH-B) and plumb a verity_hash cross-reference through disk_util + the UKI disk layout.
  • Switch UKI/systemd-boot to a slot-independent main cmdline plus per-slot verity.addon.efi addons, and switch GRUB to UUID-based systemd.verity_usr_* parameters injected at build time.
  • Add trident to the RPM package catalog and enable tridentd.socket in the built image.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
build_library/rpm/uki_install.sh Reworks UKI cmdline to be slot-agnostic and adds generation of per-slot verity addons.
build_library/rpm/rpm_install.sh Fixes punctuation in a log message.
build_library/rpm/package_catalog.yaml Adds the trident RPM to the package set.
build_library/rpm/grub.cfg Updates GRUB verity cmdline template to use injected FS/verity UUIDs instead of inline hash-offset logic.
build_library/rpm/grub_install.sh Injects FS UUID + verity UUID into GRUB template during provisioning.
build_library/rpm/build_image_util.sh Enables tridentd.socket via a sockets.target wants symlink.
build_library/disk_util Adds dps-usr-verity type resolution, supports separate hash partitions, and emits FS/verity UUIDs.
build_library/disk_layout_uki.json Adds HASH-A/HASH-B partitions and links USR-A/USR-B to them via verity_hash.
build_library/build_image_util.sh Plumbs new UUID output files into bootloader install arguments.
acl/docs/architecture.md Updates architecture docs for dedicated hash partitions / slot-agnostic verity boot.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread build_library/build_image_util.sh
Comment thread build_library/rpm/grub_install.sh
Comment thread build_library/rpm/grub_install.sh
Comment thread build_library/rpm/uki_install.sh
Comment thread build_library/rpm/uki_install.sh
Comment thread build_library/disk_util Outdated
Comment thread acl/docs/architecture.md Outdated
- grub_install.sh: define --fs_uuid/--verity_uuid so shflags accepts the
  args build_image_util passes in grub mode (was aborting the build).
- rpm/grub_install.sh: fail the build when verity is enabled but a UUID
  file is missing/empty instead of emitting an invalid empty UUID= cmdline.
- disk_util: validate blkid FS UUID output is non-empty before writing.
- rpm/uki_install.sh: correct addon header comment (mount.usr lives in the
  main UKI cmdline, not the per-slot addon).
- acl/docs/architecture.md: document the UKI PARTUUID addon boot flow.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Comment thread build_library/disk_util Outdated
Comment thread build_library/rpm/uki_install.sh Outdated
Copilot re-review follow-ups on PR #28:

- disk_util: the --fs_uuid output was only written in separate-hash-partition
  mode, so legacy inline-verity layouts produced no FS UUID file and now hard-
  fail in grub.cfg generation. Collapse the two near-identical verity branches
  into a single contextlib.ExitStack path that reads the FS UUID in both modes,
  removing the duplicated veritysetup/blkid logic.
- rpm/uki_install.sh: correct the stale UUID-validation comment (grub_install.sh
  now dies rather than warns; uki_install does not pass UUIDs through itself).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Comment thread build_library/disk_util
Comment thread build_library/rpm/grub_install.sh
When a partition's verity_hash cross-reference points to a partition number
that is not present in the loaded layout, disk_util raised a bare KeyError.
Validate the reference and raise InvalidLayout with a clear message instead,
matching how other malformed-layout cases are reported.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Comment thread build_library/rpm/uki_install.sh Outdated
Remove the unused 'count' variable from the local declaration in
_lookup_partuuid to clear shellcheck SC2034 noise. No behavior change.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Comment thread build_library/disk_util
partitions is keyed by string partition numbers (JSON object keys), and
GetPartitionByNumber() coerces with str(num) for the same reason. Normalize
hash_part_num to str right after reading it so a layout that specifies
verity_hash as a JSON number (e.g. 3 instead of "3") still resolves against
the partitions dict instead of falsely raising InvalidLayout.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- install_uki_timeout_addon: require exactly one UKI on the ESP (die on
  multiple) instead of silently picking one via sort -V | head, so the
  addon can never land in the wrong <uki>.efi.extra.d. Mirrors the
  single-UKI rule in rpm/uki_install.sh.
- RPM-mode tridentd.socket enable: hard-fail if tridentd.socket is absent.
  trident is required for ACL images, so a missing unit is a broken image;
  fail the build rather than leave a dangling wants-symlink.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 72f4b84a-8f5c-4df6-9f8e-fb4850c520e2

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Comment thread build_library/disk_util Outdated
A present-but-empty verity_hash (e.g. "verity_hash": "") previously fell
through to legacy inline-verity mode instead of separate-hash-partition
mode, producing an invalid layout that is hard to debug. Strip and raise
InvalidLayout when the value is empty, consistent with the other malformed-
layout guards in this function.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 72f4b84a-8f5c-4df6-9f8e-fb4850c520e2

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Comment thread build_library/vm_image_util.sh Outdated
Comment thread build_library/rpm/uki_install.sh Outdated
Comment thread build_library/disk_util Outdated
…ID wording

Under switch_to_strict_mode (set -e), a non-zero exit from `sudo ukify build`
aborted the script before the following existence check could emit the
intended die message or clean up the temp dir. Wrap every ukify invocation
(main UKI + firstboot/fips/kdump/debug/verity addons in uki_install.sh and
the timeout addon in vm_image_util.sh) in an explicit `if ! ...; then
rm -rf <tmp>; die; fi`, and clean up the temp dir on the produce-check path
too. Failures now surface with a clear message and no leaked temp dirs.

Also reword disk_util Verity's fs_uuid comment/exception from "btrfs FS UUID"
to "filesystem UUID": the blkid read is filesystem-agnostic. (ACL only
verities btrfs today, so this is wording-only.)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 72f4b84a-8f5c-4df6-9f8e-fb4850c520e2

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 9 comments.

Comment thread build_library/vm_image_util.sh
Comment thread build_library/rpm/uki_install.sh
Comment thread build_library/rpm/uki_install.sh
Comment thread build_library/rpm/uki_install.sh
Comment thread build_library/rpm/uki_install.sh
Comment thread build_library/rpm/uki_install.sh
Comment thread build_library/rpm/uki_install.sh
Comment thread build_library/rpm/uki_install.sh
Comment thread build_library/rpm/uki_install.sh Outdated
sudo ukify build can leave root-owned artifacts in the mktemp temp dirs, so a
plain `rm -rf` could fail and (under strict mode) abort before the intended
die. Use `sudo rm -rf` for every temp-dir cleanup in the UKI/addon builders
(uki_install.sh main UKI + firstboot/fips/kdump/debug/verity addons, and the
timeout addon in vm_image_util.sh), matching the existing sudo rm -rf
convention used throughout build_library. Each temp dir is assigned via
mktemp -d immediately before use, so the rm target is always a real path.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 72f4b84a-8f5c-4df6-9f8e-fb4850c520e2

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

GRUB's disk_layout.json previously used the legacy single-partition
inline-hash verity scheme. The UKI-side changes in this branch switched
grub.cfg's verity cmdline construction to UUID=-based hash identification
(dropping hash-offset=), which only works when the hash tree lives on its
own independently-discoverable partition.

Mirror disk_layout_uki.json's structure: add dedicated HASH-A/HASH-B
partitions (dps-usr-verity type) cross-referenced via verity_hash, so
blkid/udev can resolve systemd.verity_usr_hash=UUID=<verity_uuid> the same
way UKI does. ACL-GRUB still cannot do Trident A/B updates (no addon-swap
equivalent), but the partitioning/verity mechanics are now consistent
between the two boot paths instead of silently broken on GRUB.

ESP stays at 128MiB (no AB-staging need on GRUB). BIOS-BOOT partition is
kept (shared with Portage/flatcar builds; RPM/GRUB mode already skips
installing to it).
Brian Fjeldstad and others added 2 commits July 24, 2026 14:51
install_uki_timeout_addon already works around a CI-only QEMU-TCG flake on
arm64 kola test images: heavy parallel-VM CPU contention can prevent udev
from initialising ESP/OEM/usr-verity devices within the default initrd
device timeout, dropping the VM to an emergency shell. That fix was
UKI-only (built as a signed .efi addon dropped into <uki>.efi.extra.d/).

GRUB has no addon mechanism, so add install_grub_timeout_override: since
grub.cfg is plain text (not signed/embedded like a UKI), patch the
already-written grub.cfg copies on the test image's ESP directly via sed,
appending systemd.default_device_timeout_sec=120 to the shared
linux_cmdline. Same scoping as the UKI addon: arm64 test image only
(INJECT_DOCKER_SYSEXT=true), never applied to production images or amd64.
…ition-consistent-20260723

acl-grub: use dedicated /usr verity hash partitions consistent with UKI; fix arm64 test-image device-init timeout
Copilot AI review requested due to automatic review settings July 24, 2026 17:30

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Comment thread build_library/vm_image_util.sh
Addresses Copilot review comment on PR #28: install_grub_timeout_override
logged \ patched\ unconditionally even if the sed substitution silently
did not match (e.g. linux_cmdline format changed), leaving the timeout
override ineffective without any signal. Now verifies the substitution
took effect and dies with a clear message if not.
Copilot AI review requested due to automatic review settings July 24, 2026 18:01

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants