Skip to content

Configure remote repositories per architecture and boxes for the host - #300

Merged
daandemeyer merged 4 commits into
amutable-systems:mainfrom
daandemeyer:push-sutullnlkmoq
Oct 12, 2026
Merged

daandemeyer merged 4 commits into
amutable-systems:mainfrom
daandemeyer:push-sutullnlkmoq

Conversation

@daandemeyer

Copy link
Copy Markdown
Member

This is an alternative to #295 for #292.

The problem

Buck builds every target for a configuration: a set of facts such as "cpu is x86_64, os is linux". The
same target in two configurations is two builds, even when its output depends on only one of those facts.
Arch's core repository is reached in three configurations:

Reached by Configuration
the Arch box lock x86_64, linux
an Arch image x86_64, linux, distribution=arch
the Arch box x86_64, linux, plus the exec mark

A repository's content only depends on the cpu, so these three are the same build. Buck still runs all
three, and the three downloads of each file write to the same path at the same time. That corrupts
core.db in refresh-catalog.

What Buck offers

An exec_dep configures a tool for the execution platform, whoever depends on it. An execution platform
can also register an exec marker constraint. Buck then marks the execution configuration, so it never
equals a target configuration, even with the same cpu and os. tine copied that marker from the prelude's
example platform, and only the box's exec-only compatibility check uses it.

A rule can declare an incoming transition with rule(cfg = ...). The transition rewrites the
configuration of every caller before Buck configures the target. Callers that end up in the same
configuration share one build, and the other configurations only get a forward node that points at it.

Buck copies the exec mark through every transition. A transition can merge "x86_64, linux" with "x86_64,
linux, distribution=arch", but never with "x86_64, linux, exec".

What this changes

  1. A content-based output path contains no configuration. The x86_64 and arm64 configurations of one
    repository therefore download an identical file, such as a noarch package, to the same path. Every
    download now has the architecture in its path.
  2. The repository rule gets an incoming transition to tine//platforms:<cpu>. The box lock and the image
    then reach one repository. The box still gets its own, because of the exec mark.
  3. The execution platform stops registering the exec marker. The execution configuration is then the same
    as the default target configuration. The box rule gets an incoming transition to the host platform,
    which replaces the exec-only compatibility check. Both transitions fail with an error if an execution
    platform registers the marker again, so a consumer project cannot bring the duplicate downloads back
    without noticing.
  4. The box alias goes away. It no longer prevents a second build, but it resolved a parent's packages for
    a lock of another architecture, because a package list can select() on the architecture. A child
    box's lock now reads the parent's packages from the parent's lock for the same architecture. A child
    box can therefore only keep a lock for an architecture its parent keeps one for.

Compared with #295, there are no hidden per-architecture targets and no aliases, and tools/catalog.py does
not change. A repository or box named on the command line reaches the same build as every other caller.

Testing

Check Result
tools:check on each commit passes
box example tests and examples/image:reproducibility-test pass, image digests unchanged
fedora.rawhide.box.lock.arm64 resolved on an x86_64 host identical to the committed lock
a child of tine//catalog:fedora.rawhide.box in another package, with locks configures
clean build of tine//catalog:arch.rolling.box.lock.x86_64 and tine//examples/image:base.arch succeeds and downloads 210 MiB, the same as #295

Fixes #292

A content-based output path contains no configuration. The x86_64 and arm64
configurations of one repository therefore download an identical file, such
as a noarch package, to the same path. A build for arm64 on an x86_64 host
needs both configurations, and two parallel downloads into one path corrupt
the file. Put the architecture into the path of every download.

Signed-off-by: Daan De Meyer <daan@amutable.com>
A box lock and an image reach a remote repository in configurations that also
differ in other constraints, such as the image's distribution. Each of those
configurations downloads every file of the repository again. Give the
repository rule an incoming transition to the target platform of its
architecture, so that callers of one architecture share one repository.
Callers in the execution configuration still get a repository of their own.

Signed-off-by: Daan De Meyer <daan@amutable.com>
Buck keeps the exec mark through transitions, so a remote repository is
configured once for the box that installs from it and once more for the box
lock and the image. The two configured repositories download the same files at
the same time, and the parallel downloads corrupt Arch's core.db in
refresh-catalog. Register no exec marker, so that the execution configuration
is the host platform's. Configure a box for the host with an incoming
transition instead of the exec-only compatibility check.

Signed-off-by: Daan De Meyer <daan@amutable.com>
The box rule's transition configures every box for the host, so the alias no
longer prevents a second build. The alias still resolves a parent's packages
for a lock of another architecture, because packages can select() on the
architecture. Let a child box's lock read the parent's packages from the
parent's lock for the same architecture instead. A child box can then only
keep a lock for an architecture that its parent keeps one for.

Signed-off-by: Daan De Meyer <daan@amutable.com>

@martinpitt martinpitt left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@daandemeyer
daandemeyer merged commit 3d65783 into amutable-systems:main Oct 12, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

refresh-catalog is failing

2 participants