image: Add description file and hook to write release metadata

[pothos]

From an artifact output one had to list the files and know by convention
how each type of file is named and what part of the filename means what.
This required the consumer to reimplement parts of the naming logic and
stay in sync. Also, the consumer might want to add more files but that
would require a second artifact dir or generating files out of band,
e.g., for signed release metadata.

Add a artifacts_.json file with a simple format to state the kind
of a listed file, the image it belongs to, and the verity hash. Add an
"author" hook that runs a given command on that artifacts file and can
produce more artifacts for us, part of the single artifacts output dir.
The consumer can now work with roles instead of file names and generate
other metadata based on that. With author_local_only it can read host
signing key without storing something in the cache.

[martinpitt]

Thanks! I had an initial conceptual/strategic look, sorry for the delay. I didn't look at the code details too much, let's figure out the broad strokes first.

[martinpitt]

This is a too strong claim.

kind: "sbom" above isn't unique. These have subtargets [sbom][cyclonedx] and [sbom][spdx], resulting in image_0_x86-64.cdx.json and image_0_x86-64.spdx.json.

The same is true for "listing", there's four of them. There's also the separate .initrd SBOMs, etc.

The commit message says "had to list the files and know by convention how each type of file is named", which sounds like you try to change our TUF generation to only look at this metadata, and not the file names any more? That wouldn't be true for at least the cases above. I think the example below is reduced to the point where all of these cases don't appear any more?

So this depends on your goal: if this file should be the only one the TUF generator looks at, it needs to contain more metadata/subtypes.

[pothos]

It's okay if the author hook refines a kind more, the point is that it doesn't have to construct a careful regex that prevents matching all other possible naming schemes to find out the right kind to refine:

_SUPPLEMENT_TYPES = {
    ".spdx.json": "application/spdx+json",
    ".cdx.json": "application/vnd.cyclonedx+json",
}

def supplement_type(name: str, entry: Entry) -> dict[str, str]:
    if entry["kind"] == "sbom":
        for suffix, media in _SUPPLEMENT_TYPES.items():
            if name.endswith(suffix):
                return {"type": media}
    if entry["kind"] == "pkgdb":
        return {"type": "application/vnd.amutable.pkgdb"}
    return {}

The kind avoid the tedious duplication of naming scheme knowledge.
If you want we can make this a bit more specific, maybe sbom:spdx or sbom.spdx?

[martinpitt]

This sentence needs some humanization/clarification.

[martinpitt]

So "author" is a project defined post-processing hook that massages the output files further? Maybe that should be called build or something else.

For such a hook, it may be tempting to edit/clean up the passed artifacts.json and the --files tree. But it must never do that: these files live in buck-out/, and hand-editing stuff there is "here be dragons".

Conceptually: this hook lives under the control of the built branch, so once we become SLSA conformant, it must not have access to the TUF signing keys. This hence can only output an unsigned release artifact, and somehow the calling infra needs to rebuild/verify it. How does that work? Calling this step from the main branch in a built tree sounds not well defined. My gut feeling is that this has to either live outside of the consuming repo, or (to eat our cake and keep it too), the whole author script lives in the consuming repo, but works outside of buck and the infra will always call it from main only. WDYT?

[pothos]

Maybe authorhook?
Good point that this is not about edits, only additions and this should be documented (and we could even assert that it didn't modify anything, good idea?).

About the concept: The SLSA signing requirement is about the signing of provenance only, but here this author hook produces an additional artifact that like the rest gets covered by the SLSA provenance envelope and signed by the SLSA key. So now the question is when we cross a trust boundary and the one who imports the release with the additional artifact checks the SLSA statement and signature on behalf of the final client, does it delegate trust to the key the builder on the other side of the boundary had access to already? Then the solution is to not use the same key but a temporary one and resign. The hook could still offer two or three modes: signed with the final key because it's a dev build or no trust boundary way crossed and the build input verified upfront, or signed with a non-final key and then resigning happens, or unsigned because it gets signed again (and to get there is sent over a trusted transfer channel that does not need a signature). This is policy the hook and/or the caller/consumer of the build system have to take care of.

[martinpitt]

Note that the action still gets cached locally in buck-out/. So the author script is still only being run once, unless the inputs change. This is probably marginal, i.e. that wouldn't re-run when the signing key changes -- that's acceptable.

[martinpitt]

That "lets it" reads like setting the flag would give the build any additional privileges. It doesn't, this needs to be implemented in the caller logic. Perhaps "That makes it safe to ...".

But still, I have a lot of Bauchschmerzen with this approach honestly, see the SLSA concerns above.

[martinpitt]

This needs to be checked for collisions. artifacts_x86-64.json might already be a target that the consuming OS itself creates, and this would overwrite it.

[martinpitt]

The file moved to docs/user/

[martinpitt]

Please use util.fail()

[martinpitt]

The listed files here don't have digests. systemd-sysupdate needs a SHA256SUMS[.gpg], but this commit neither writes such a file nor can it be derived from the artifacts file. That also ties in with the above discussion of "who/where does this get verified?" What's the direction here?