Skip to content

Reuse build comparison baselines and summarize CI mismatches - #17

Open
AdamZ-8113 wants to merge 15 commits into
Paliak:tests-branchfrom
AdamZ-8113:ci-simple-cache
Open

AdamZ-8113 wants to merge 15 commits into
Paliak:tests-branchfrom
AdamZ-8113:ci-simple-cache

Conversation

@AdamZ-8113

@AdamZ-8113 AdamZ-8113 commented Sep 11, 2026

Copy link
Copy Markdown

Description of the problem being solved:

This updates the existing build tests to reuse one calculated base across PRs and makes the comparison results easier to review on GitHub. I worked through it locally and in my fork using the live pob.codes API.

Each run downloads the corpus once and uses that same collection on both sides. The decoded builds and calculated base stay in GitHub Actions cache, keyed by the base commit, corpus contents, and test/runtime inputs. A matching cache means only the PR's builds need calculating. After the base changes, the next run calculates a new baseline automatically if one hasn't already been prepared. There's also a nightly job to prepare it ahead of time; PRs don't depend on that job having run.

The existing batches of 50, PoB restarts between batches, saved-XML comparison, and timing comparison remain. Calculated-stat differences now make the check fail after all builds have been compared. A short job summary shows affected-build counts and before/after examples, with the full report still available for digging further. XML and timing differences remain informational. A completed baseline is still cached when the comparison finds mismatches.

This uses the official PoB test image and Lua. It fixes the local cache permissions and includes the startup/export/test-mock fixes needed to run the current sources. The headless callback removal is the same fix as #16. There is no new Python dependency, custom Dockerfile, downloaded build collection in Git, or change to PoB's calculation formulas or coverage settings. The intentional errors used in the test PRs aren't part of this branch.

The full diff against tests-branch is 13 files, 420 additions and 226 deletions. The workflow continues to target dev; this PR targets tests-branch for review.

Steps taken to verify a working solution:

  • Test PR #4: numerical mismatch and summary uses the latest implementation. I deliberately doubled the Life granted by Strength in one calculation line. The GitHub run calculated and compared all 500 API builds plus five fixtures on both revisions, then failed on the stat differences. It reported 414 affected builds, including 351 with changed Life. Dual Savior went from 92 to 118 Life. The summary step, baseline cache save, and full report upload all completed successfully despite the red check. Sign in to GitHub and open the run's Summary page to see the short report.
  • Test PR #1: unchanged calculations and cache reuse completed comparisons with a comment-only change. The cold run calculated 505 builds on each side; the cached run calculated 505 PR builds and reused all base results.
  • Test PR #2: calculation failure and recovery deliberately stopped calculations and failed CI. After merging PR Fix other related issues and add support for a couple mods. #1 into the isolated test branch, I removed that error. The recovery run passed and calculated both sets of 505 builds against the new base, checking the fallback after a merge.
  • Locally, all 590 unit tests pass, and Actionlint validates the workflow. With the latest implementation, an unchanged six-build comparison passed with zero stat differences. The same comparison with the intentional Strength mistake completed all six builds and failed on five affected builds. Earlier local runs also exercised full 505-build comparisons, cache reuse, and cache invalidation when the corpus or test tools change.

PRs #1 and #2 were run before the new summary and fail-on-stat-differences behavior was added. Their green results demonstrate execution, caching, and recovery; some stat differences were still present. Existing output variation can also make the stricter check red. This PR doesn't add tolerances or fix those calculation issues.

Automatic nightly triggering is still unverified. Manual baseline preparation and reuse passed, but I didn't observe an actual scheduled run during the test window. The signed-in summary rendering also wasn't visually checked; its generated contents and successful summary step were verified. The fork tests use an isolated base branch and one-day artifact retention; those settings aren't included here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants