[WIP] Support immutable hosts and configurable installation paths - #815
Draft
Philip Lombardi (plombardi89) wants to merge 4 commits into
Draft
Philip Lombardi (plombardi89) wants to merge 4 commits into
Philip Lombardi (plombardi89) wants to merge 4 commits into
Conversation
The agent writes its own host-side files to hard-coded paths under /usr/local: the daemon binaries and their blue-green links, the nspawn lifecycle helper, the daemon recovery script, and the LocalDNS network helper. On a host with a read-only /usr none of those writes can succeed, so the agent cannot be installed at all. Add AgentConfig.HostPrefix and a resolver that derives the host-side layout from it. Paths inside the nspawn machine are untouched: they are relative and joined with the machine directory, and conflating the two would break every host. The prefix is declared, never inferred. Where the agent may write is a property of the filesystem, not of the distribution, so keying on distro identity would misclassify a hardened host with a read-only /usr and would silently relocate files on any host whose os-release changed. A wrong guess is expensive to recover from, because the lifecycle helper path is baked as an absolute path into the nspawn drop-in and the config regeneration unit. The accepted syntax is narrow on purpose. The prefix is interpolated into generated systemd units and into a shell script, neither of which quotes it, so rather than adding two kinds of escaping that every consumer must keep correct, the value is constrained to be inert in both. Teardown and existing-deployment detection need to sweep both the configured prefix and the default, so that changing the prefix cannot orphan files or let a dirty host be silently reprovisioned; KnownHostPrefixes and MergeHostPrefixes exist for that and are used by the callers that follow. Nothing consumes the resolver yet. This is the model and its validation, so the changes that convert each caller can be read on their own. Hosts that do not set a prefix resolve to exactly the paths they had before, pinned by a regression test against the existing constants.
Philip Lombardi (plombardi89)
force-pushed
the
acl-extract/immutable-host-paths
branch
from
September 19, 2026 02:29
e56881a to
364cb10
Compare
The blue-green agent binaries were absolute constants under /usr/local/bin. A host whose /usr is read-only cannot hold them there, which is the whole reason the prefix exists. ResolvedAgentUpgradePathsFor resolves them under a prefix instead. An empty prefix selects the default, and a test pins that the result is exactly the constants this package used before, because those paths are baked into generated units and into the blue-green symlinks of every host already installed. If the default drifted, an upgraded agent would look for its binaries where the host does not have them. The original entry point stays, deprecated, delegating to an empty prefix. It is published from pkg/ and callers outside this repository compose their own phases from it, so removing it would break them at compile time. Every caller inside the repository moves to the new one in this commit, because staticcheck's SA1019 is enabled and a split would not lint. All of them run under systemd or on the host with no config in hand, so they take the prefix from the applied config, which is what that lookup exists for. On a host that configures no prefix this resolves the default and nothing changes. The AgentUpgrade signal path is deliberately not prefixed: it is state about an upgrade rather than part of the installed layout, and it already lives under the agent config directory, which stays writable on such hosts. One caller passed the function as a value rather than calling it, so a search for call sites missed it and only the linter found it. It is now wrapped, so the prefix is read when the command runs rather than when it is constructed.
Teardown has to find the agent's own files. On a host that configures a prefix they are not under /usr/local, and after a bootstrap that failed before the node started there is nothing on the host that says where they are: the applied config carries the prefix but is not written until the node runs. The ownership record is written before any mutation, which makes it the only source that covers that window, so it carries the resolved prefix. Optional, and the schema version does not move. A record written by an agent that knows about the prefix stays readable by one that does not, because unknown fields are ignored, and a default installation writes no field at all so its record is byte-identical to one written before this existed. A test pins that, since the value of the compatibility is entirely in the absence. Resolved rather than configured, so the record names a real directory instead of an empty string meaning whatever the default happened to be. NewRecord takes it as a parameter rather than leaving it a field to set afterwards. Forgetting it would be silent and would only surface at teardown, on a host whose files are somewhere reset does not look. Also corrects a comment in the prefix lookup that pointed at this field before it existed.
The agent's own binaries live under the prefix, so starting with a different one is not a retry of the same installation. Continuing would leave the first installation's files where they are and build a second one beside them. Admission has to refuse and ask for a reset, which is what a changed fingerprint does. The delicate half is the other one. Every host already installed was fingerprinted without this input. If the default contributed a value, all of them would hash differently under an agent carrying this change, read as a different installation, and demand an explicit reset on upgrade over a field they never set. So the prefix enters the hash only when it resolves somewhere other than the default, and carries omitempty so that at the default it contributes nothing rather than an empty string. It is the resolved prefix that counts, not how it was written. Leaving it unset and naming /usr/local explicitly put the files in the same place, so they hash alike; telling an operator who wrote down what was already true that they must reset the host would be a poor trade for the precision. Verified by mutation, since all three ways to get this wrong are silent and affect every host in the field rather than the one under test: dropping omitempty, hashing the default instead of eliding it, and never hashing the prefix at all each fail a test. The fixtures carry a literal fingerprint, which is what makes the first two detectable at all.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Status: WIP. Do not merge. Depends on #742.
This PR targets
acl-extract/bootstrap-recovery(#742), notmain, and that isdeliberate. It needs the installation ownership package #742 introduces, so it
cannot build against
mainuntil #742 lands. Basing it on the branch keeps thisdiff to its own commit rather than re-proposing #742's.
After #742 merges this PR will look wrong until it is rebased. The repository
squash-merges, so #742 will land as a single commit that does not match the
commits on its branch. GitHub will auto-retarget this PR to
main, and the diffwill then re-propose all of #742's changes against a
mainthat already containsthem. The fix is a rebase onto main from #742's final head, not a merge:
Why
The agent writes its own host-side files to hard-coded paths under
/usr/local:the daemon binaries and their blue-green links, the nspawn lifecycle helper, the
daemon recovery script, and the LocalDNS network helper. On a host with a
read-only
/usrnone of those writes can succeed, so the agent cannot beinstalled at all.
What is here so far
AgentConfig.HostPrefix, andgoalstates.ResolveHostPathsderiving the agent'shost-side layout from it. Paths inside the nspawn machine are untouched: they are
relative and joined with the machine directory, and conflating the two would
break every host.
The prefix is declared, never inferred. Where the agent may write is a property
of the filesystem, not of the distribution, so keying on distro identity would
misclassify a hardened host with a read-only
/usrand would silently relocatefiles on any host whose os-release changed. A wrong guess is expensive to recover
from, because the lifecycle helper path is baked as an absolute path into the
nspawn drop-in and the config regeneration unit.
The accepted syntax is narrow on purpose. The prefix is interpolated into
generated systemd units and into a shell script, neither of which quotes it, so
rather than adding two kinds of escaping that every consumer must keep correct,
the value is constrained to be inert in both.
The daemon binaries resolve under it.
ResolvedAgentUpgradePathsForplaces theblue-green binaries and their links beneath the prefix; the original entry point
stays, deprecated, delegating to an empty prefix, and every caller in this
repository moved in the same commit because
staticcheckreports SA1019 againstour own calls. The AgentUpgrade signal path is deliberately not prefixed: it is
state about an upgrade rather than part of the installed layout.
The ownership record carries the resolved prefix. It is written before the first
host mutation, which makes it the only thing that knows where this installation
put its files after a bootstrap that failed before the node started; the applied
config carries the same value but does not exist until then. The field is
optional and the schema version does not move, so a record written by an agent
that knows about the prefix stays readable by one that does not.
Bootstrap identity includes it, but only when it resolves somewhere other than
the default. Changing the prefix is not a retry of the same installation, so it
must be refused rather than continued; leaving it unset must hash exactly as it
did before the field existed, or every host already installed would demand a
reset on upgrade over a field it never set. Identity follows where the files
land rather than how the prefix was spelled, so naming
/usr/localexplicitlyand leaving it unset agree.
The remaining consumers are listed below.
Compatibility
Hosts that set no prefix resolve to exactly the paths they had before, pinned by
a regression test against the existing constants. That matters because the
lifecycle helper path is baked as an absolute path into generated systemd units
that are already on disk.
Still to come
nspawn lifecycle helper and LocalDNS. The installer still hard-codes
/usr/local/bin/unbounded-agentas its install target.and the default, so changing the prefix cannot orphan files or let a dirty host
be silently reprovisioned.
KnownHostPrefixesandMergeHostPrefixesexistfor this and have no callers yet.
Until those land the prefix is accepted and validated but not honoured
everywhere, which is why this stays a draft: a host that sets it would still
have some files placed under
/usr/local.Not in this change
No Ignition delivery and no persistent repave redesign. Script and manual
installation must remain a complete supported path on an image that has the
required prerequisites.