Add JUnit report generation support to test runner - #72
Conversation
The progress output prints one character per step, which says how many scenarios failed but not which ones. That makes it hard to compare two runs, such as a run against a WordPress build under test and a run against the build it is based on, in order to tell which failures the change under test actually caused. Setting `WP_CLI_TEST_JUNIT_DIR` now additionally writes JUnit reports to the given directory. Those name every scenario along with the file and line it is defined on. Failed scenarios are run a second time. Reports of that rerun go to a `rerun` subdirectory, as they only cover the scenarios that failed the first time and would otherwise overwrite the full report of the package. Keeping both makes it possible to tell a flaky scenario, which fails once and then passes, from one that consistently fails. The progress output is unchanged, and nothing happens unless the environment variable is set. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CE81GsxUY597AdaMP1NXzk
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
Included review availability: Your plan includes up to 1 review per rolling hour; 0 remain after this review. 📝 WalkthroughWalkthroughThe test script now supports optional JUnit reports for initial runs and failed-scenario reruns. It removes stale reports, creates separate report locations, and logs the configured destination. The README documents configuration, naming, storage, and flaky-scenario reporting. ChangesJUnit reporting
Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk: ⚪ Minimal · up to The change adds optional JUnit report output while preserving existing test-runner behavior when the setting is absent; no actionable merge-blocking risk remains beyond normal checks. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Adds optional JUnit XML report generation for Behat-based test runs to improve CI/CD integration and result tracking, while keeping existing progress output behavior.
Changes:
- Documented a new
WP_CLI_TEST_JUNIT_DIRenvironment variable and the report/rerun directory structure inREADME.md. - Updated
bin/test-sourceto emit JUnit reports (including rerun reports under arerun/subdirectory) whenWP_CLI_TEST_JUNIT_DIRis set.
Reviewed changes
Copilot reviewed 1 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| README.md | Adds usage docs and explains how initial vs rerun JUnit reports are organized. |
| bin/test-source | Adds conditional Behat CLI arguments and directory creation to generate JUnit XML reports (plus rerun reports). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@bin/test-source`:
- Around line 55-61: Update the JUnit setup around JUNIT_ARGS and
JUNIT_RERUN_ARGS to remove stale XML reports owned by this runner before the
test loop begins, including prior files under the rerun directory. Preserve the
existing output paths and avoid deleting unrelated files in
WP_CLI_TEST_JUNIT_DIR.
- Around line 17-21: The flaky-detection documentation must use testcase status
transitions: in bin/test-source lines 17-21, rewrite the shell comments to
compare matching testcase statuses, identifying failed-then-passed as flaky; in
README.md line 49, document failed-to-passed as flaky and failed-to-failed as
consistently failing.
In `@README.md`:
- Line 44: Update the report-path code block in README.md to declare the text
language by changing its opening fence to ```text, while preserving the listed
paths and closing fence.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: e6e97d94-1ff3-4a58-9fa6-e43cc4cf342c
📒 Files selected for processing (2)
README.mdbin/test-source
The rerun report holds a result for every scenario it ran, including the ones that passed the second time, rather than omitting them as the documentation claimed. A flaky scenario is therefore one reported as failed in the report of the package and as passed in the rerun report, not one that is missing from the latter. Reports of an earlier run are now removed before the tests start. A package that passes produces no rerun report at all, so a report left behind by an earlier run would have been read as belonging to the current one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CE81GsxUY597AdaMP1NXzk
|
@coderabbitai review |
✅ Action performedReview finished.
|
Summary
This change adds support for generating JUnit XML reports from Behat test runs, enabling better integration with CI/CD systems and test result tracking.
Key Changes
$WP_CLI_TEST_JUNIT_DIRenvironment variable support to optionally write JUnit reports to a specified directoryrerunsubdirectory to distinguish between flaky and truly failing testsImplementation Details
$WP_CLI_TEST_JUNIT_DIRis set, the script creates the directory structure (includingrerunsubdirectory) and constructs appropriate Behat format arguments--format junit --out std --out <dir>pattern is used to maintain progress output on stdout while writing JUnit XML to the specified directoryhttps://claude.ai/code/session_01CE81GsxUY597AdaMP1NXzk
Summary by CodeRabbit
New Features
Documentation