Reuse build comparison baselines and summarize CI mismatches - #17
Open
AdamZ-8113 wants to merge 15 commits into
Open
AdamZ-8113 wants to merge 15 commits into
AdamZ-8113 wants to merge 15 commits into
Conversation
…OfBuildingCommunity#15) Co-authored-by: Paliak <91493239+Paliak@users.noreply.github.com>
Paliak
force-pushed
the
tests-branch
branch
from
September 13, 2026 09:53
4221c26 to
501d036
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of the problem being solved:
This updates the existing build tests to reuse one calculated base across PRs and makes the comparison results easier to review on GitHub. I worked through it locally and in my fork using the live pob.codes API.
Each run downloads the corpus once and uses that same collection on both sides. The decoded builds and calculated base stay in GitHub Actions cache, keyed by the base commit, corpus contents, and test/runtime inputs. A matching cache means only the PR's builds need calculating. After the base changes, the next run calculates a new baseline automatically if one hasn't already been prepared. There's also a nightly job to prepare it ahead of time; PRs don't depend on that job having run.
The existing batches of 50, PoB restarts between batches, saved-XML comparison, and timing comparison remain. Calculated-stat differences now make the check fail after all builds have been compared. A short job summary shows affected-build counts and before/after examples, with the full report still available for digging further. XML and timing differences remain informational. A completed baseline is still cached when the comparison finds mismatches.
This uses the official PoB test image and Lua. It fixes the local cache permissions and includes the startup/export/test-mock fixes needed to run the current sources. The headless callback removal is the same fix as #16. There is no new Python dependency, custom Dockerfile, downloaded build collection in Git, or change to PoB's calculation formulas or coverage settings. The intentional errors used in the test PRs aren't part of this branch.
The full diff against
tests-branchis 13 files, 420 additions and 226 deletions. The workflow continues to targetdev; this PR targetstests-branchfor review.Steps taken to verify a working solution:
PRs #1 and #2 were run before the new summary and fail-on-stat-differences behavior was added. Their green results demonstrate execution, caching, and recovery; some stat differences were still present. Existing output variation can also make the stricter check red. This PR doesn't add tolerances or fix those calculation issues.
Automatic nightly triggering is still unverified. Manual baseline preparation and reuse passed, but I didn't observe an actual scheduled run during the test window. The signed-in summary rendering also wasn't visually checked; its generated contents and successful summary step were verified. The fork tests use an isolated base branch and one-day artifact retention; those settings aren't included here.