Report host provisioning capability truthfully - #814
Open
Philip Lombardi (plombardi89) wants to merge 4 commits into
Open
Philip Lombardi (plombardi89) wants to merge 4 commits into
Philip Lombardi (plombardi89) wants to merge 4 commits into
Conversation
Azure Container Linux reports ID=azurelinux with VERSION_ID=3.0.x, so it matched both the azurelinux case and the RPM-family fallback and was silently classified as mutable Azure Linux 3. They are different operating systems. ACL is a Flatcar-derived immutable image whose /usr is a read-only dm-verity mount and which ships no usable package manager, so rpm is not merely absent from PATH, it does not exist. Classify it before both paths, keyed on VARIANT_ID=azurecontainerlinux. ID_LIKE containing flatcar is a deliberate second signal so an image that later drops VARIANT_ID does not regress into being treated as mutable Azure Linux 3; no Azure Linux 3 image advertises it. The azurelinux ID is required alongside it so a plain Flatcar host is not claimed. The resolved rootfs is unchanged. The nspawn rootfs runs on the host kernel rather than supplying its own, and the Azure Linux 3 image is built from the same base, so it stays ABI-correct for both. Nothing branches on this yet. It is the predicate the next change needs to decide whether host prerequisites can be installed or only verified, which is why HostDistroIsImageManaged is introduced here with it. Tests use the verbatim os-release from the published ACL image and cover each signal alone, quoted values, and that plain Flatcar is not claimed. A regression case asserts real Azure Linux 3 still resolves to azlinux3.
InstallPackages is the first bootstrap task, and it aborted on any host without apt-get, tdnf or dnf. That is right for a host merely missing a package manager and wrong for one that has none by design: an immutable image can already provide curl, nft, mountpoint and systemd-nspawn while shipping nothing to install with. Fall back to probing the capability each required package exists to provide, and accept the host when every one resolves. Keyed on the capability rather than the distribution deliberately, so it covers any immutable image without enumerating them. Image-managed hosts take a further step. Azure Container Linux ships tdnf, so keying only on "is there a package manager binary" selected tdnf and then attempted installation into a read-only dm-verity /usr. There the image is the unit of delivery, so a missing tool is a prerequisite to report, and the error names the tool and points at the image or a system extension rather than claiming a package manager is required, which would be false with tdnf sitting right there. The rpm query gains the same fallback. An RPM host is not guaranteed to ship the rpm binary: Azure Container Linux has a populated rpm database but only rpm-libs, so rpm -q exits 127 and every package looks missing, sending bootstrap across the network to install packages that are already there. Package manager preference is unchanged. A host with a working rpm still queries rpm, and a real package manager still wins over the capability path so a genuinely missing package can be remediated. A capability-only host has nothing to install with, so InstallPackages guards against a nil command rather than dereferencing it should a tool disappear after detection. The four existing detection tests now call detectHostPackageManagerFor rather than detectHostPackageManager, because the latter reads the host's real /etc/os-release to decide whether the host is image-managed, and a unit test must not depend on the machine running it.
Preflight verified the sysctl and systemd unit directories were writable but never checked where the agent installs its own files. A host with a read-only /usr passed every check and then failed partway through bootstrap, when installing the bootstrap binary hit EROFS with no indication that the prefix was the problem. Probe it during preflight instead, as an error rather than a warning: this is not a degraded mode, it is the first thing bootstrap does to the host, so it would fail there anyway with a worse message. The probe asks whether the directory can be created, not whether it exists. The agent creates it during bootstrap, so on a host that has never been bootstrapped it is absent by definition and probing it directly with os.CreateTemp would return ENOENT and refuse a perfectly good host. Walking up to the nearest existing ancestor asks the question the check means. The rootfs preflight already did that walk with its own nearestExistingParent, so it moves to utilio as NearestExistingDir and both callers use it rather than growing a second copy. The checked directory is derived from DaemonBinaryPath rather than restated, so the check cannot drift from where the agent actually installs. The list is passed in rather than computed inline so a later change can supply a configured location without restructuring the checker.
The flush was ordered only against systemd-nspawn@.service, which is enough during bootstrap but not across a reboot. A host that enables its own firewall starts it and the flush within milliseconds of each other with no ordering between them, and when the firewall lands last it restores an INPUT policy of drop. The failure that produces is quiet, which is what makes it worth ordering explicitly. The kubelet's own connection is outbound, so the node still reports Ready, while every inbound request to the kubelet port is dropped and kubectl logs and exec time out against a node that looks healthy. These units do not exist on the Debian hosts or the RPM cloud images, and After= on an absent unit is a no-op, so nothing else changes.
Jason Wilder (jwilder)
approved these changes
Sep 17, 2026
hbc (bcho)
approved these changes
Sep 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make host preflight tell the truth about whether a host can be provisioned,
before anything mutates it. The agent stops inferring what it can install from
which package manager binary happens to exist, and starts checking whether the
tools it needs are actually usable and whether it can write where it installs.
Why
InstallPackagesis the first bootstrap task, and it decides what to do byprobing for
apt-get, thentdnf, thendnf, taking the first that resolvesand aborting if none do. Two things are wrong with that.
A host with no package manager is refused even when it already has every tool
the agent needs. Nothing is missing and nothing needs installing, but bootstrap
stops anyway.
Worse, the presence of a package manager binary is not evidence that packages
can be installed. An image-based host can ship
tdnfwhile mounting/usrread-only, so detection selects it and then attempts installation into a
filesystem that cannot accept it. The same assumption appears in the
rpmquery: a host can have a populated rpm database and no
/usr/bin/rpm, whererpm -qexits 127, every required package looks missing, and bootstrap reachesacross the network to install packages that are already present.
Preflight had a matching gap. It verified the sysctl and systemd unit
directories were writable but never checked where the agent installs its own
binaries, so a host with a read-only
/usrpassed every check and then failedpartway through bootstrap with an EROFS that named no cause.
What changes
executable it exists to provide:
systemd-containertosystemd-nspawn,curltocurl,nftablestonft,util-linuxtomountpoint. When nopackage manager resolves, the host is accepted if every capability does. The
rpmquery falls back the same way when the binary is absent. Package managerpreference is unchanged, so a genuinely missing package is still remediated.
delivered as an image, a missing tool is a prerequisite to report. The error
names the tool and points at the image or a system extension, instead of
claiming a package manager is required, which would be false with
tdnfsitting right there.
ID=azurelinuxwith a 3.x
VERSION_ID, so it previously matched mutable Azure Linux 3 andwas treated as one.
VARIANT_IDis the authoritative signal andID_LIKE=flatcaris a deliberate second one, so an image that later dropsVARIANT_IDdoes not silently regress. The resolved rootfs is unchanged.be created, not whether it exists, because the agent creates it during
bootstrap and it is absent by definition on a host that has never been
bootstrapped. The
nearestExistingParentwalk the rootfs preflight alreadyhad moves to
utilio.NearestExistingDirso both callers share it.enables its own firewall can apply it after the flush, restoring an INPUT
policy of drop. The node still reports Ready because the kubelet's connection
is outbound, while every inbound request to the kubelet port is dropped.
Compatibility
Hosts with a working package manager behave exactly as before: the same manager
is selected, with the same refresh and install arguments. The capability path is
reached only when no package manager resolves.
Two behaviors do change on existing hosts. A host that previously aborted at
InstallPackagesnow proceeds if every capability resolves, and an RPM hostwithout the
rpmbinary now reports packages as present rather than missing.Both are the intended correction. The preflight install-directory probe is a new
way for preflight to fail, on hosts where bootstrap could not have succeeded.
Not in this change
This does not make Azure Container Linux work. It is a precondition, because
bootstrap otherwise stops at the first task, but installing there also requires
a writable location for the agent's own files, which is a separate change. No
configurable install prefix, no Ignition delivery, and no change to how the
agent installs on any currently supported host.