Create 0316-launcher-decoupling

[Hemant28codes]

No description provided.

[buildpack-bot]

Maintainers,

As you review this RFC please queue up issues to be created using the following commands:

/queue-issue <repo> "<title>" [labels]...
/unqueue-issue <uid>

Issues

(none)

[edmorley]

which is necessary for certain environments or security configurations

Hi! Could you expand a bit on this? It seems to be the key point behind the motivation here, but currently is a bit vague?

I'm guessing the reason is so that app image rebases can pick up new launcher versions for patching Go vulns reported in the binary by security scanners, or something similar?

[Hemant28codes]

Yes that's the main reason, I will update the motivation section.

[Hemant28codes]

Updated the motivation section.

[dmikusa]

Could this be an argument to rebase instead? I like the idea of being able to update this, but I'm not a fan of stuffing more things into our run images.

Ex: pack rebase --lifecycle <new lifecycle> where that's a version, image ref, etc.. like we do with run-image

[Hemant28codes]

We don't use pack rebase to swap run images, we directly use lifecycle for builds. If this is an argument to rebase, it won't solve the case when someone is not using pack rebase to swap run image layers

[Hemant28codes]

"stuffing more things into our run images." - This will be totally optional, as the launcher binary needs to be part of run image, and as the new run image is available it should contain the upgraded launcher.

[Hemant28codes]

"pack rebase --lifecycle " - we don't want to upgrade the entire lifecycle as part of this, because in our builder we have lifecycle installed, which is then used to create the final app image by its creator binary. The final export phase which adds the /cnb/lifecycle/launcher is what we need to have it configurable and pointing through run image.

[dmikusa]

We don't use pack rebase to swap run images, we directly use lifecycle for builds. If this is an argument to rebase, it won't solve the case when someone is not using pack rebase to swap run image layers

I don't follow your use case then. If you're rebuilding, then won't the lifecycle just include the most recent launcher? Why do you need to have it take something different from a different location?

"stuffing more things into our run images." - This will be totally optional, as the launcher binary needs to be part of run image, and as the new run image is available it should contain the upgraded launcher.

Yes, but I want to use this feature. 😄 It's a good feature and something I know will address needs people have voiced to me.

There are a lot of pack build and pack rebase users, so I think this proposal should include them as well. Requiring them to customize a run image is going to be a non-starter for a lot of users. Passing a flag to pack rebase, much easier.

[dmikusa]

"pack rebase --lifecycle " - we don't want to upgrade the entire lifecycle as part of this, because in our builder we have lifecycle installed, which is then used to create the final app image by its creator binary. The final export phase which adds the /cnb/lifecycle/launcher is what we need to have it configurable and pointing through run image.

If lifecycle is what's adding the launcher in your case, then perhaps we could add an argument to lifecycle to specify where it loads launcher?

[Hemant28codes]

"I don't follow your use case then. If you're rebuilding, then won't the lifecycle just include the most recent launcher? Why do you need to have it take something different from a different location?" - we are not rebuilding, we build the container once, but instead of using pack rebase, we have our mechanism of swapping the run image by some layer annotations.

[Hemant28codes]

As a part of proposal, it can be extended to pack rebase as well, but only adding it in the pack, will not cover the case for users directly using lifecycle to build the container.

[dmikusa]

we are not rebuilding, we build the container once, but instead of using pack rebase, we have our mechanism of swapping the run image by some layer annotations.

That's fine, but what you're talking about is a bespoke solution, whereas many people use pack rebase. If the proposal covers both cases, I'm 100% in. This is something that will help a lot.

[Hemant28codes]

Sure we can have both the usecases covered.

Just to give you an idea of what our usecase is:

• Current Setup

Source code + Builder(contains lifecycle) -----(Build)----> Application image(contains launcher from lifecycle in builder + run image)

• Proposed Setup

Source code + Builder(contains lifecycle) -----(Build)----> Application image( run image), Run image -> will have launcher installed

One question here:
Can we have different versions for lifecycle building the application and the lifecycle version from which we are downloading the launcher?

[maigovannon]

Should this be a boolean or is it better to pass in a path to the launcher? It shouldn't matter whether its in the Run image or Build image as long the users of the pack/lifecycle decide to create that launcher at some point. So in effect is it better to have --launcher-path=<my_launcher>path and the equivalent CNB_LAUNCHER_PATH? @dmikusa FYI

[Hemant28codes]

Actually yes this makes sense.

[jjbustamante]

Thanks for this RFC — the use case is real. Security scanners flagging launcher CVEs that force full rebuilds across thousands of app images is genuinely painful, and worth solving.

After the working group discussion I dug into the lifecycle code a bit, and I think there's a different mechanism that gets to the same goal without changing the OCI artifact structure.

The problem with the current proposal

Skipping the launcher layer at build time (--launcher-in-run-image) produces a structurally different OCI artifact — fewer layers, different manifest digest, different ordering. This creates a divergent artifact problem: images built with and without the flag are structurally incompatible. It also makes rebase ambiguous when an image is missing a launcher layer — which was flagged at the end of the working group meeting.

What the metadata already gives us

The io.buildpacks.lifecycle.metadata label already tracks the launcher layer digest:

{ "launcher": { "sha": "sha256:3e4529cbb..." } }

The rebaser already reads this label in full. It knows exactly which layer is the launcher. It just doesn't use that today beyond bookkeeping.

A different approach: extend rebase with --launcher-image

Instead of changing the build artifact, extend the rebase phase with an optional --launcher-image:

lifecycle rebase \
  --run-image new-run:v2 \
  --launcher-image buildpacksio/launcher:0.20.1 \
  app-image:latest

The rebaser uses origMetadata.Launcher.SHA to locate the launcher layer, replaces it with the binary from --launcher-image, updates the SHA in the metadata, done. Same number of layers, same positions, same layer names — only the bytes change.

This covers both workflows

For pack rebase users — straightforward, just a new flag.

For the annotation-based workflow you described — these are independent operations. Your existing run image swap stays untouched. You'd only call lifecycle rebase --launcher-image separately when a launcher CVE hits. No platform changes needed.

Custom launchers also work: platforms already use --launcher-path at build time to inject their binary, and they'd use --launcher-image my-org/launcher:patched at rebase time to patch it.

What this needs

1. A buildpacksio/launcher:<version> OCI artifact published alongside buildpacksio/lifecycle:<version>
2. --launcher-image flag on the rebase phase (new Platform API version)
3. imgutil support for replacing a specific layer by digest — the current Rebase() only does bottom-up replacement, this would be a mid-stack swap

Happy to help think through the spec language or the imgutil side if this direction makes sense to the group.

[Hemant28codes]

@jjbustamante the approach you suggested, won't this require running lifecycle rebase --launcher-image for each application image built?

This is our current architecture -

If you see this architecture, after building the container once, we actually only have filesystem storing it, and at the serving time just pulling all the layers and run the clone. So "lifecycle rebase --launcher-image" running this everytime is kind of impossible in this case for every user.

Can we maybe have something like we have some annotation at the time of building it itsleff which actually creates a symlink to the run image, so whenever we see that there is a new launcher that will be updated in the run image, and symlink will start pointing to that.
To have the same OCI structure, can we instead of including the cnb/lifecycle/launcher through builder in final application image, we instead have the one in our run image and the layer in which cnb/lifecycle/launcher stays that actually is empty and only a symlink?

[jjbustamante]

Hey Hemant, to answer your question directly — yes, the --launcher-image approach I suggested would require running lifecycle rebase --launcher-image for each application image individually. At Google's scale, that's simply not viable, and your architecture diagram makes that very clear: you're storing layers separately in a filesystem and swapping the run image layer once for all images simultaneously. Running a per-image operation defeats the whole point.

Your diagram is great to understand the problem. One thing I'd suggest making very explicit in the RFC — and this came up in the working group discussion — is that the "Buildpacks Application Launcher" layer must always be present in the application image, regardless of whether the feature is used or not. The OCI artifact structure stays identical in both cases. Today that layer carries a 2.8 MB self-contained executable at /cnb/lifecycle/launcher (you can see this in any CNB-built app image using a tool like dive).

With this feature, the layer still exists in the same position, same name — but instead of the binary, it carries a symlink. It would be good to include a concrete before/after in the RFC showing this difference explicitly.

What changes is that the launcher layer now has an explicit dependency on the run image layer. That dependency needs to be front and center in the RFC, because it has real implications for the rebaser that need to be explored.

For the rebaser: today, rebase validates OS/arch compatibility and run image identity — nothing more. With symlink-based launcher layers, the rebaser gains a new responsibility: it must verify that the new run image actually provides a launcher binary at the expected path before completing the rebase. Without this, a successful rebase can produce an image that silently fails at runtime when /cnb/lifecycle/launcher resolves to a broken symlink. To support this validation, io.buildpacks.lifecycle.metadata would need to record whether the launcher is symlink-based and what path it resolves to, and run images carrying a launcher should advertise it via a label.

This RFC is opening the door to fix a real problem for platforms operating at high scale — the ability to patch launcher CVEs without triggering full rebuilds across thousands of images. That's a meaningful contribution. For the broader community, TOC and maintainers — the --launcher-image rebase approach might address a different audience (smaller platforms using standard pack rebase). Is that worth covering in this RFC or better tracked as a separate proposal?

[jjbustamante]

One more thing worth calling out — RFC #333 ("Rebase Buildpack Contributed Layers") by @jabrown85 is working on a related problem: extending rebase to selectively patch any buildpack-contributed layer using a metadata file and a patch OCI image.

The --launcher-image approach I mentioned in my earlier comment is actually a specific instance of what #333 proposes more generally. If #333 moves forward, its mechanism could cover the launcher patching case for platforms using standard pack rebase — without needing a separate flag in #336.

That actually strengthens the case for this RFC to focus narrowly on what #333 can't solve: the high-scale, annotation-based swap architecture where a per-image rebase operation isn't viable, and the launcher needs to be a dependency of the run image rather than a self-contained layer.

Both RFCs will also share infrastructure work — specifically, imgutil needs to support mid-stack layer replacement by digest, which neither can do today. Worth coordinating there.

It might be useful to cross-reference both RFCs and make explicit which use cases each one is meant to address.

[dmikusa]

@jjbustamante

The --launcher-image approach I mentioned in my earlier comment is actually a specific instance of what #333 proposes more generally. If #333 moves forward, its mechanism could cover the launcher patching case for platforms using standard pack rebase — without needing a separate flag in #336.

That works for me, so long as this use case is capture on #333. This would be the approach used by probably 99% of the Paketo users.

With this feature, the layer still exists in the same position, same name — but instead of the binary, it carries a symlink. It would be good to include a concrete before/after in the RFC showing this difference explicitly.

What changes is that the launcher layer now has an explicit dependency on the run image layer. That dependency needs to be front and center in the RFC, because it has real implications for the rebaser that need to be explored.

Does this mean that if I don't want this new symlink functionality, then there's no change to how things work presently? Since this is being added for a very specific subset of users/use cases, I just want to make sure this is an opt-in feature.

[Hemant28codes]

Hi @jjbustamante I have modified the RFC to include the specific details mentioned in your above comment.

[jjbustamante]

@dmikusa

Does this mean that if I don't want this new symlink functionality, then there's no change to how things work presently? Since this is being added for a very specific subset of users/use cases, I just want to make sure this is an opt-in feature.

If you don't want to use the symlink approach with the launcher in the run-image, you will be able to use the approach in #333 or maybe the idea I suggest, which I think could be a feature in pack that uses the implementation of #333

[jjbustamante]

@Hemant28codes could you rename the file to text/0316-launcher-decoupling.md ?

[Hemant28codes]

Two questions I have in mind:

• Can we have a custom launcher or do we need to depend on the launcher provided by buildpacks, or maybe add on top of the buildpacks launcher?
• Will the launcher going to be a seperate binary which doesn't depend on spec - buildpack API, platform API, lifecycle version, libcnb version and will be decoupled from these, since there will be applications which are built with certain versions of specifications (buildpacks API, platform API, libcnb version, lifecycle version), but when the launcher gets rebased it should not have any dependency on these and not break?

[jjbustamante]

Two questions I have in mind:

• Can we have a custom launcher or do we need to depend on the launcher provided by buildpacks, or maybe add on top of the buildpacks launcher?

yes, this works. The run image vendor controls what binary sits at the spec-defined launcher path, so it can be the standard CNB launcher, a custom variant, or CNB-plus-extras. This is actually something you can already do at build time with the lifecycle's -launcher flag, it's a lifecycle-side flag, not exposed by pack, so platforms like yours that invoke the lifecycle directly have always had it available. The new mode in this RFC essentially moves that injection from build-time into the run image, so a single run image update covers all apps.

[jjbustamante]

@Hemant28codes

• Will the launcher going to be a seperate binary which doesn't depend on spec - buildpack API, platform API, lifecycle version, libcnb version and will be decoupled from these, since there will be applications which are built with certain versions of specifications (buildpacks API, platform API, libcnb version, lifecycle version), but when the launcher gets rebased it should not have any dependency on these and not break?

I do not think the launcher will stop depending on the spec, the whole idea of the spec is to define the rules and then a Platform implementor like Google, could replace it with their own binary but satisfying the spec.

What IS achievable is a compatibility window. A newer launcher can run an older-built app as long as its supported API version range covers the version recorded in the app metadata. libcnb, you're right about — that's only used by buildpacks to write layer metadata, the launcher doesn't interact with it, so no coupling there. Lifecycle version isn't directly relevant either — what matters is the Platform API and Buildpack API versions the launcher supports.

Implications for the RFC that I think should be called out explicitly:

1. The launcher in the run image MUST support the Platform API and Buildpack API versions recorded in the app image metadata — otherwise the app silently breaks at runtime
2. The rebaser needs to validate this before swapping — it has the app's API versions in metadata.toml, and the new run image should advertise what API versions its launcher supports
3. The run image label advertising the launcher should include the supported API version ranges, e.g.:

io.buildpacks.run-image.launcher.platform-apis = "0.7,0.8,...,0.15"                                                                                                                       
io.buildpacks.run-image.launcher.buildpack-apis = "0.7,0.8,...,0.11"                                                                                                                      

5. Over time, as old APIs get deprecated and new launchers drop support for them, there will be a point where an old app image can no longer have its launcher updated without also
   rebuilding. That's fine — it just needs to be an explicit part of the compatibility model.

I understand, from a Google perspective, you will be rebasing the run image and probably skipping the validations because at that scale, it is hard to do, but these implications are more around CNB and how to guarantee things will not break for others.

[Hemant28codes]

Oh okk make sense @jjbustamante

[Hemant28codes]

@jjbustamante is it possible to set buildpack API version at build time in image environment varriables?
To support custom launcher it needs to know the buildpack and platform API version at runtime to invoke the corresponding launcher satisfying the requirements?

[jabrown85]

The launcher in the run image MUST support the Platform API and Buildpack API versions recorded in the app image metadata

This right here is where things get a bit tricky. A run image does not support just one app. Without the build process placing a launcher layer that is known to be compatible with the app being built that just went through all the build and validations - there is no real paved path to making sure produced app images can always successfully run. That indeterminism makes me a bit uncomfortable.

If you are the ones doing the building and the running it works fairly well. You are in complete control. But if you are producing builders for the community this gets a lot more messy IMO. How long will the run-image provided launcher support Platform API X.XX? If end users are rebasing weekly - will they caught out by the sudden symlink change of the launcher and now their previously 2 yr old working app is not launchable?

@jjbustamante is it possible to set buildpack API version at build time in image environment varriables?
To support custom launcher it needs to know the buildpack and platform API version at runtime to invoke the corresponding launcher satisfying the requirements?

No, because each buildpack may target different buildpack APIs. The metadata in the image itself has the information on the buildpack APIs.

I'll give this some more thought this week if I can find time

[jabrown85]

@Hemant28codes I believe you said your platform doesn't really store the entire produced image after a buildpacks build. Is there a reason you don't only store the App Layers? If you are already omitting the base layers so you can stitch them together on the platform to execute the App Image, can you also stop storing the launcher and mount https://hub.docker.com/r/buildpacksio/lifecycle into /cnb like you do the base layers? That way your platform dictates the lifecycle/launcher without it being in the artifact tied to the build.

I'd also like to get your thoughts on producing a much smaller launcher. The scanning tools often pick up the launcher layer because it is go based. The CVEs that pop are almost entirely false for launcher but we patch anyway. If you instead bundled a much smaller surface area version of launcher - would that satisfy your needs without complicating distribution of app images? I'd be willing to try and transform launcher into a rust version, like @hone did once in the past.

[Hemant28codes]

Hi @jabrown85 "can you also stop storing the launcher and mount https://hub.docker.com/r/buildpacksio/lifecycle into /cnb like you do the base layers" - for this part, will the issue of compatibility with which platform and buildpack API version the image is built still remain?

If this is not the case and the launcher you are suggesting is independent of platform and buildpack API, then can't we just have it in the run image itslef, in that way also it won't be tied to build.

"How long will the run-image provided launcher support Platform API X.XX "- For this we are planning to have a go binary which will act as a meta launcher - containing various launcher (let's say one corresponding to each major version), and based on platform api version and buildpack api version present in launcher it will invoke the appropriate launcher.

[Hemant28codes]

The suggestion to produce a much smaller launcher is a great idea and would certainly be beneficial for reducing the scan surface area. That being said, we believe the proposal to decouple the launcher and allow it to be provided by the run image offers a significant additional advantage. The primary edge of this approach is the ability to update the launcher in running applications without requiring a full rebuilds. I think giving the ability to users to allow custom launchers decoupled from lifecycle will help regardless of launcher .