Cache dependencies and build outputs to improve workflow execution time

[wroyca]

The workflow execution times can be extremely slow due to dependencies being reinstalled and build outputs being rebuilt on each run. To improve workflow execution times, we should add caching functionality to the build2 CI action, which would store previously installed dependencies and build outputs in a cache. When running subsequent builds, the action could then retrieve these cached artifacts instead of rebuilding them from scratch.

Reference documentation for caching dependencies in GitHub Actions workflows can be found here: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows

[Rookfighter]

True, I was thinking about the same some time ago, though there are a few open points:

• I am not entirely sure if we can refer to other Actions (like actions/cache@v3) from within a custom GitHub action. The examples in your link only refer to use-cases in Workflows. I also could not find any examples for doing that. Do have experience with that?
• According to the docs: We would need a full a list of all paths to all files that were installed by the build2 installer (or better: a way to generate the list), so we can provide these files to the cache. @boris-kolpackov maybe build2 already provides such a functionality?
• We need a smart way to uniquely identify which cache entry should be used (key generation). The toolchain.sha256 file (see here) contains a unique version of build2 in the first line, which can be used. However we still need to distinguish, which OS we are currently running on (Ubuntu 20.04, 22.04, Windows 10, Windows 11, MacOS XX) and which toolchain it is using (MSVC 19, 22; Clang 12, 13 ,14; GCC 7, 8) and also encode that into the caching key. For compiling on other architectures than amd-64 (e.g. ARM or RISC V) we would also need to consider the target architecture. Note: I know build2 provides this kind of information, but we need to extract this information before build2 is installed (for cache lookup).

It might be easier to provide an example on how users can use the caching Action in their repos themselves to speed up build2 CI times, rather than integrating it into the build2 action. Repo local caching would require less sophisticated key handling and thus generates less complexity. However, there would be some redundancy, because essentially every repo would want to use caching.
One more point is also acceptance by new users: If the build2 Action requires super long build times (without additional effort to implement caching), interested new users might lose interest / motivation at some point. So in the long-term it might be beneficial to have lower / faster CI times out of the box.

[boris-kolpackov]

Hm, to me it's a bit unclear what is suggested to be cached: the build2 toolchain itself, the dependencies of the package being built, or both?

We would need a full a list of all paths to all files that were installed by the build2 installer (or better: a way to generate the list), so we can provide these files to the cache. @boris-kolpackov maybe build2 already provides such a functionality?

Not yet, but sounds like the same as/related to build2/build2#133.

[wroyca]

Hm, to me it's a bit unclear what is suggested to be cached: the build2 toolchain itself, the dependencies of the package being built, or both?

We would need a full a list of all paths to all files that were installed by the build2 installer (or better: a way to generate the list), so we can provide these files to the cache. @boris-kolpackov maybe build2 already provides such a functionality?

Not yet, but sounds like the same as/related to build2/build2#133.

I'd say both actually, but more about build2 itself as we don't have prebuilt availables

[Rookfighter]

Since this issue is raised on the repo for GitHub Action for build2, I assumed you inteded to cache the build2 setup step.

@boris-kolpackov builds on the GitHub CI currently usually take around 15-40 min (depending on underlying OS and GitHub execution speed), from which usually < 2min are spent actually building and testing the package in the repo. Most of the time is spent building / installing build2, so it would make sense to cache the installed build2 artifacts.

@wroyca IMO I would not see caching dependencies or other artifacts from the build process as part of this action. I think that must be handled either by the build system (which is not part of the build2 design goals as far as I understood it) or by the package maintainer for their respective project / repo. But this is just my opinion, not any official roadmap, so correct me if my understanding is wrong, @boris-kolpackov

[wroyca]

Since this issue is raised on the repo for GitHub Action for build2, I assumed you inteded to cache the build2 setup step.

It was to my understanding that the distribution process, that is fetching from cppget as opposed to building dependencies, could be considered as setting up build2 for a particular projects - Otherwise, that's correct. There is some ambiguity in that regards. 🙂

[Klaim]

I also setup (again) a workflow in a new repo yesterday and hit the same lack of caching of build2's install issue, so here I am. 🥲

So if I understand correctly, the cache identifier would be platform + build2's version + ... and that's it? Because usually we dont really care how build2 itself is being built (except if working on it) but we do care that it's ready to work with any toolchain we want it to use. I believe nobody uses the default toolchain in CIs but to use it when trying a toy example.

The build2's version is necessary and we need to make sure that stage versions are handled correctly, although that could also mean one cache per stage version, which is a maybe a lot and will not hold with the way the caching system is working (although there is 10Go of cache).

For compiling on other architectures than amd-64 (e.g. ARM or RISC V) we would also need to consider the target architecture.

If I understand correctly, that's not really important here because that have to be provided to build2 in the configurations, it's not something you need to provide to build build2 itself, isnt it?

It might be easier to provide an example on how users can use the caching Action in their repos themselves to speed up build2 CI times, rather than integrating it into the build2 action

As long as the example is simple, that would be a good enough solution for me too.

And I'm in the same situation where I don't understand enough the cache system to set it up myself, I'm trying to find someone who could help but nobody aruond me use that (they all use the one in gitlab that seems simpler).

[boris-kolpackov]

So if I understand correctly, the cache identifier would be platform + build2's version + ... and that's it?

If you look inside the install scripts/batch files, they keep track of quite a few versions and checksums. We could add an option to print one of those (for example, build2-toolchain archive checksum). We also detect the platform but on POSIX we do it by calling config.guess inside build2-toolchain, which means we will have to first download and unpack it, something that you would probably want to avoid in this case.

on the GitHub CI currently usually take around 15-40 min (depending on underlying OS and GitHub execution speed), from which usually < 2min are spent actually building and testing the package in the repo.

Wow, that's very slow. On my workstation local build2 bootstrap take under 2 minutes. Also, I doubt the <2min will apply to more complex packages, like, say Qt, which take substantially longer than build2 to build on my machine.

I think that must be handled either by the build system (which is not part of the build2 design goals as far as I understood it) or by the package maintainer for their respective project / repo. But this is just my opinion, not any official roadmap, so correct me if my understanding is wrong, @boris-kolpackov

There are some building blocks that could potentially be used to assemble something that maybe solves this (emphasis on "potentially" and "maybe"). For example, right now we are working on both consuming and producing system packages (currently for Debian/Fedora and alike). So theoretically one could use such pre-built system packages to satisfy dependencies. Also, there are plans for distributed compilation and caching but that will probably work on the translation unit basis and where such a cache would run in this case is unclear.

[Klaim]

on the GitHub CI currently usually take around 15-40 min (depending on underlying OS and GitHub execution speed), from which usually < 2min are spent actually building and testing the package in the repo.

Wow, that's very slow. On my workstation local build2 bootstrap take under 2 minutes. Also, I doubt the <2min will apply to more complex packages, like, say Qt, which take substantially longer than build2 to build on my machine.

I can confirm, for example on this recent project of mine where I am setting up the CI, I observe 18'30'' to setup build2 alone on ubuntu, 32'50'' on Windows.
Worse: the performance can widely vary! For example for the same project a previous ci build but using the same build2 setup action took 25'32'' on ubuntu, and the steps failed later, but the setup succeeded and it was the same github-action in both cases (shorter time and longer time).
And yeah the time to run the build2 script locally is far shorter for me too, although I have a 12 core system so that helps.

Github-Actions is cool and it's free...so yeah it's also quite slow in general and you're not guaranteed any performance consistency. I think they assumed people would optimize their CI by just gathering binaries and building only the local project, which is kind of the usual way people setup CI builds outside big companies that rebuild everything and share translation cache. Also a lot of people are using docker to handle this, which is kind of a system image caching system in some ways. (personally I would like to avoid having to work with docker in my small projects)

[boris-kolpackov]

Github-Actions is cool and it's free...so yeah it's also quite slow in general and you're not guaranteed any performance consistency.

Our (as in, build2 project's) CI service is also free and while we don't guaranteed any performance consistency, we do try to keep it reasonable and it's currently definitely faster than GitHub. Plus we do cache the build2 toolchain build (more precisely, we bootstrap a CI VM once and then use its snapshots for builds). So the question is why not use our CI service instead? Is there some integration missing that we could add?

[Klaim]

On the top of my head:

• the main missing thing is integration with github's pull-request system: most projects around here are setup so that PRs are not mergeable before the CI is all green, which implies that the PR system knows how to invoke the ci (whatever it is, not necessarilly github-actions) -that is doable- , then receive the results -that is the part missing I believe?- and is able to use the results both to display the status and block the merge of PRs. One could argue that you can do all this manually, but it's a major pain to do it that way when the project starts having some regular participation.
• no way to control the (system) dependencies outside the ones used through build2 - but I believe you are working on features to help with that? Anyway if the dependency in question needs to be gathered by other means than package managers (example: Vulkan dev kit implementations), we end up with the same issue and need for custom ci setup.
• no way to control how the environment is setup, notably for potential environment variables, special directories being used etc. - but that's mainly an issue for interactive executables and testing the ux of these, IMO.
• no way to add custom steps which are not tests of a target - testscript can be exploited to do that but it's not exactly the same thing as running arbitrary code in the environment (like github-actions does).

The first two points are the main reasons to have a github-action (or gitlab etc.). The second point is why we setup imgui package with github-action first, if you remember.
When it's applications, in particular graphic non-cli ones (including games), the two last points can quickly become relevant.
If we ignore the case of applications, which might not be much relevant for build2's ci, and focus on libraries, then only the 2 first points are important. (clarification: the project I pointed is a game project, that's why I didnt even consider using build2's ci, it's not a library and not something I'll distribute through cppget)

Also if there was a way to have github-actions invoke build2's ci, then wait and retrieve the results and some of the generated artefacts, then most if not all of these points would be solved. Not sure it would be useful though, in case of need to debug it seems complicated.

[boris-kolpackov]

the main missing thing is integration with github's pull-request system [...] then receive the results -that is the part missing I believe?

Right, there is even a relevant issue: build2/build2#15 I think if there is an integration point in GitHub for that, I would imagine something that polls for result availability which is returned in some structured form (hopefully JSON), then we could definitely look into supporting this in brep.

no way to control the (system) dependencies outside the ones used through build2 [...]

Yes, there is an ongoing work on supporting system package managers but something ad hoc like Vulkan dev kit will still be an issue. But see the next point, perhaps this could be the solution.

no way to control how the environment is setup [...]
no way to add custom steps which are not tests of a target - testscript can be exploited to do that but it's not exactly the same thing as running arbitrary code in the environment

Yes, we've been thinking about that, most recently in the context of uploading distribution packages. One idea is to factor the scripting language used in Testscript and Buildscript into something standalone and allow writing custom actions in that.

[Klaim]

Right, there is even a relevant issue: build2/build2#15 I think if there is an integration point in GitHub for that, I would imagine something that polls for result availability which is returned in some structured form (hopefully JSON), then we could definitely look into supporting this in brep.

I totally forgot that issue haha, anyway yes solving this makes build2 ci an obvious choice at least when making libraries, maybe in general.

Yes, we've been thinking about that, most recently in the context of uploading distribution packages. One idea is to factor the scripting language used in Testscript and Buildscript into something standalone and allow writing custom actions in that.

Indeed that might help. I'm not 100% sure it would be enough but that would widen the cases where build2's ci is a good choice.

To get back to the specific topic of this issue, the more I look at it, the more I think providing an example to the users of a way to cache the install of build2 is the way to go. I don't think it can be done adequately for every use case of this github-action in a generic way. However so far I failed to setup such example, I need more time to be able to play with this cache system. If someone else manage to get a good working example feel free to post here.

[Rookfighter]

Our (as in, build2 project's) CI service is also free and while we don't guaranteed any performance consistency, we do try to keep it reasonable and it's currently definitely faster than GitHub. Plus we do cache the build2 toolchain build (more precisely, we bootstrap a CI VM once and then use its snapshots for builds). So the question is why not use our CI service instead? Is there some integration missing that we could add?

TBH I love the bdep ci, the pipelines are setup with a lot of attention for details and provide a high quality feedback on submitted packages. They handle many corner cases that I would never have thought of and as hobbyist I could never reach such a degree of detail with my GitHub workflows. So it actually helps a lot to increase the quality of packages by just reading the warning / failure reasons.

I usually use a combination of GitHub Workflows and bdep ci. Mainly GitHub Workflows are just convenient, so I use them out of the following reasons:

• Integration with GitHub: automatic start on push and pull request, explicit association with commits (first failed / succeeded commit)
• Automation: Builds are automatically triggered and I get automatic notifications when the build has finished. I do not have to poll the results. Also I do not have to remind contributors on PRs that they should post a bdep ci link after every commit.
• Installing custom system dependencies

In a perfect world, I would love to have a GitHub action which triggers builds on bdep ci (on push and pull requests), then polls the created builds and reports the results. And actually that shouldn't be that hard to implement (assuming that you could give us some insight on the backend API of bdep ci). However, there's the chicken egg problem, that I need to have bdep available on the github runner to trigger bdep ci. So we would be back to caching the build2 binaries.

[Rookfighter]

For compiling on other architectures than amd-64 (e.g. ARM or RISC V) we would also need to consider the target architecture.

If I understand correctly, that's not really important here because that have to be provided to build2 in the configurations, it's not something you need to provide to build build2 itself, isnt it?

Maybe I was looking a bit too far into the future, but architecture might be relevant when your runner (your host platform) is ARM (or something else). For example GitHub supports hosting runners on Apple Silicon (ARM64) see here, so the cache should not load x86 MacOS binaries, because they would not work. But this might be a less relevant issue at this point.

[Rookfighter]

To get back to the specific topic of this issue, the more I look at it, the more I think providing an example to the users of a way to cache the install of build2 is the way to go. I don't think it can be done adequately for every use case of this github-action in a generic way. However so far I failed to setup such example, I need more time to be able to play with this cache system. If someone else manage to get a good working example feel free to post here.

Same here. I might take a look at it at some point, but not any time soon I guess.