[amjjbonvin]

Before submitting, read the contributing guidelines and the AI policy.

What does this PR do and why?

This PR adds examples/run_examples_regression.py, which replaces the manual run-and-compare procedure for the integration tests. It runs all examples/*/*-test.cfg files on the current code and compares the results with a reference built from origin/main.

• Reference: init builds it in a separate git worktree, so the developer's checkout is not touched.
• Skipped: MPI and batch examples, and follow-up configs that need an existing run (--extend-run, --restart). 55 tests are run.
• Completion check: haddock3 must exit cleanly, the log must end with Finished at and contain no ERROR lines, and every module must have its step folder. The number of output models and the files written by each step are compared with the reference. This also covers the scoring and analysis examples that have no caprieval step.
• CAPRI statistics, per caprieval step:
  • number of acceptable, medium and high quality models (DockQ criteria, plus il-RMSD criteria for protein-ligand);
  • rank of the first acceptable model;
  • rank and quality of the best model.
• Report: examples/regression/report.md lists new failures, output differences, quality and ranking changes with better/worse markers, runtime changes, and skipped tests.

Housekeeping:

• Removed: run_tests.py, compare_runs.py and run_examples-full.py, and the test that kept the first two in sync. The two scripts used a hard-coded list of 30 tests.
• Added: the extract-capri-stats-*.sh scripts, which summarise a single run.
• Docs: rewrote examples/README.md and docs/pages/testing/integration_tests.md for the new workflow.

How was this tested?

• tests/test_examples_regression.py covers test selection and skips, CAPRI statistics (DockQ and ligand criteria), the completion check, and the comparison (changes, new failures, failures on both sides).
• I checked the statistics against extract-capri-stats-dockq.sh and extract-capri-stats-ligand.sh on existing runs; the values are identical.
• Full init + run (55 tests each) comparing main with the cns-scripts-consistency branch. All tests ran, and the report flagged the expected differences in the CG examples.
• I manually used it to check the results of a new branch. All working fine. Nice and concise report

AI assistance

Claude Code wrote the script, the tests and the documentation updates based on my specifications and checked them as described above. I reviewed the changes and ran the full regression set myself.

Checklist

• Tests cover the new and/or changed code
• Documentation updated if needed (also in the haddock3 user-manual) (developer tool, documented in docs/ and examples/README.md; no user-manual change needed)
• CHANGELOG.md updated for user-facing changes

Related issues

Closes #1731

Notes for reviewers

• A full run takes about 3 hours per branch locally, so this is not meant for the CI.
• Existing run directories of the examples are deleted before each run (--list shows which).
• Runtime comparisons are only meaningful if both runs used the same machine under similar load.

🤖 Generated with Claude Code

[Comp-era]

When init --only is used, the new results are merged into the existing reference.json, but the top-level commit field is overwritten without checking whether the previous results came from the same commit.

Since each init fetches the latest origin/main, rerunning only a subset of tests after main has advanced can produce a reference containing results from different commits, while the report identifies only the latest commit. The same applies to current.json if the branch gets new commits between run --only calls.

Could we add a check that compares the current commit with the one stored in the JSON file before running the tests? If they differ, the script could require a full reference rebuild or allow the user to specify the previous commit using --ref-branch. Alternatively, storing the commit for each test would allow the report to identify mixed reference results.

[amjjbonvin]

Thanks @Comp-era, good catch. Addressed in 374cdf6 with a combination of both suggestions:

• Commit per test: each test result now records the commit it was run on.
• Mixed merges refused: init --only / run --only stop before running if the saved results that would be kept come from another commit. The message suggests rerunning without --only, using --ref-branch <commit> (for init) or checking out the stored commit (for run). Only the hash is compared, so a "(+ local changes)" note does not trigger it.
• --allow-mixed overrides the check; the report header then lists the tests that come from other commits.

Older result files without per-test commits fall back to the file's commit. A test covers the check, the override and the report header, and the docs describe the new behaviour.

[Comp-era]

I pulled the latest changes and verified the fix. All tests passed, and the commit-consistency check correctly prevents partial runs from mixing results from different commits.

Looks good from my side.