From 89675285f85489a99e16ce7bcb24e50f464bc0ed Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 9 Aug 2026 16:00:08 +0000 Subject: [PATCH 1/2] Optionally write JUnit reports The progress output prints one character per step, which says how many scenarios failed but not which ones. That makes it hard to compare two runs, such as a run against a WordPress build under test and a run against the build it is based on, in order to tell which failures the change under test actually caused. Setting `WP_CLI_TEST_JUNIT_DIR` now additionally writes JUnit reports to the given directory. Those name every scenario along with the file and line it is defined on. Failed scenarios are run a second time. Reports of that rerun go to a `rerun` subdirectory, as they only cover the scenarios that failed the first time and would otherwise overwrite the full report of the package. Keeping both makes it possible to tell a flaky scenario, which fails once and then passes, from one that consistently fails. The progress output is unchanged, and nothing happens unless the environment variable is set. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CE81GsxUY597AdaMP1NXzk --- README.md | 17 +++++++++++++++++ bin/test-source | 28 ++++++++++++++++++++++++++-- 2 files changed, 43 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 5e046c06d..f8e403f55 100644 --- a/README.md +++ b/README.md @@ -31,6 +31,23 @@ The following options can be set: * **`stable `** - Use the latest stable phar release. * **`all`** - Use both the latest stable release phar as well as the nightly phar. +### JUnit reports + +`$WP_CLI_TEST_JUNIT_DIR` writes JUnit reports to the given directory, in addition to the progress output that is always printed. This is useful to compare the results of two runs, as the reports name every scenario along with the file and line it is defined on, which the progress output does not. + +```bash +WP_CLI_TEST_JUNIT_DIR=build/junit TEST_PACKAGE=wp-cli/entity-command composer test +``` + +Behat names each report after the suite it belongs to, so one package results in one report: + +``` +build/junit/wp_cli_entity_command.xml +build/junit/rerun/wp_cli_entity_command.xml +``` + +Failed scenarios are run a second time. Reports of that rerun are written to the `rerun` subdirectory, as they only cover the scenarios that failed the first time and would otherwise overwrite the full report of the package. A scenario that is reported as failed in the first report but does not appear in the rerun report passed on the second attempt, and was therefore flaky rather than failing. + ### Automated Builds This repository is being rebuilt through a Travis CI cron job every 24 hours to post test results in Emails and Slack. diff --git a/bin/test-source b/bin/test-source index 204690c94..a3af69a9b 100755 --- a/bin/test-source +++ b/bin/test-source @@ -9,6 +9,16 @@ # - "all": The framework as well as all bundled commands are tested. # - "commands": Only the command packages are tested. # - : Only the package named is tested. +# +# $WP_CLI_TEST_JUNIT_DIR optionally writes JUnit reports to the given directory, +# in addition to the progress output. Behat names each report after the suite it +# belongs to, so one package results in one report. +# +# Reports of the rerun of the failed scenarios are written to a "rerun" +# subdirectory, as they only cover the scenarios that failed the first time and +# would otherwise overwrite the full report of the package. A scenario that is +# reported as failed in the first report but does not appear in the rerun report +# passed on the second attempt, and was therefore flaky rather than failing. set -e @@ -39,6 +49,20 @@ if [ "$TEST_PACKAGE" != "all" -a "$TEST_PACKAGE" != "commands" ]; then REPOS="$TEST_PACKAGE" fi +JUNIT_ARGS=() +JUNIT_RERUN_ARGS=() + +if [ -n "$WP_CLI_TEST_JUNIT_DIR" ]; then + mkdir -p "${WP_CLI_TEST_JUNIT_DIR}/rerun" + + # The first --out belongs to the progress format and keeps it on stdout, the + # second one is the directory the JUnit reports are written to. + JUNIT_ARGS=(--format junit --out std --out "${WP_CLI_TEST_JUNIT_DIR}") + JUNIT_RERUN_ARGS=(--format junit --out std --out "${WP_CLI_TEST_JUNIT_DIR}/rerun") + + echo "Writing JUnit reports to ${WP_CLI_TEST_JUNIT_DIR}" +fi + for REPO in $REPOS; do echo "Testing package $REPO..." @@ -47,9 +71,9 @@ for REPO in $REPOS; do echo "Behat Tags: $BEHAT_TAGS" set +e - "${BUILD_DIR}/vendor/bin/behat" --format progress $BEHAT_TAGS --strict --suite $REPO + "${BUILD_DIR}/vendor/bin/behat" --format progress "${JUNIT_ARGS[@]}" $BEHAT_TAGS --strict --suite $REPO if [ $? -ne 0 ]; then - "${BUILD_DIR}/vendor/bin/behat" --format progress $BEHAT_TAGS --strict --suite $REPO --rerun + "${BUILD_DIR}/vendor/bin/behat" --format progress "${JUNIT_RERUN_ARGS[@]}" $BEHAT_TAGS --strict --suite $REPO --rerun if [ $? -ne 0 ]; then FAILED_PACKAGES="$FAILED_PACKAGES ${RELEASE}:${REPO}" fi From 2762685aa221d4f8d6585094e7d0db681be71dae Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 9 Aug 2026 16:26:29 +0000 Subject: [PATCH 2/2] Correct the flaky scenario semantics and clear stale reports The rerun report holds a result for every scenario it ran, including the ones that passed the second time, rather than omitting them as the documentation claimed. A flaky scenario is therefore one reported as failed in the report of the package and as passed in the rerun report, not one that is missing from the latter. Reports of an earlier run are now removed before the tests start. A package that passes produces no rerun report at all, so a report left behind by an earlier run would have been read as belonging to the current one. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CE81GsxUY597AdaMP1NXzk --- README.md | 11 +++++++++-- bin/test-source | 10 +++++++--- 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index f8e403f55..406d70df3 100644 --- a/README.md +++ b/README.md @@ -41,12 +41,19 @@ WP_CLI_TEST_JUNIT_DIR=build/junit TEST_PACKAGE=wp-cli/entity-command composer te Behat names each report after the suite it belongs to, so one package results in one report: -``` +```text build/junit/wp_cli_entity_command.xml build/junit/rerun/wp_cli_entity_command.xml ``` -Failed scenarios are run a second time. Reports of that rerun are written to the `rerun` subdirectory, as they only cover the scenarios that failed the first time and would otherwise overwrite the full report of the package. A scenario that is reported as failed in the first report but does not appear in the rerun report passed on the second attempt, and was therefore flaky rather than failing. +Failed scenarios are run a second time. Reports of that rerun are written to the `rerun` subdirectory, as they only cover the scenarios that failed the first time and would otherwise overwrite the full report of the package. + +The rerun report holds a result for every scenario it ran, which is what tells a flaky scenario from a failing one: + +* A scenario reported as `failed` in the report of the package and as `passed` in the rerun report failed once and then passed, and was therefore flaky. +* A scenario reported as `failed` in both is failing consistently. + +Reports left behind by an earlier run are removed before the tests start, so that the directory only ever describes the run that wrote it. A package that passes produces no rerun report at all, which would otherwise leave a stale one in place. ### Automated Builds diff --git a/bin/test-source b/bin/test-source index a3af69a9b..b18f1eea7 100755 --- a/bin/test-source +++ b/bin/test-source @@ -16,9 +16,9 @@ # # Reports of the rerun of the failed scenarios are written to a "rerun" # subdirectory, as they only cover the scenarios that failed the first time and -# would otherwise overwrite the full report of the package. A scenario that is -# reported as failed in the first report but does not appear in the rerun report -# passed on the second attempt, and was therefore flaky rather than failing. +# would otherwise overwrite the full report of the package. The rerun report +# holds a result for every scenario it ran: one reported as passed there was +# flaky, one reported as failed is failing consistently. set -e @@ -53,6 +53,10 @@ JUNIT_ARGS=() JUNIT_RERUN_ARGS=() if [ -n "$WP_CLI_TEST_JUNIT_DIR" ]; then + # A package that passes does not produce a rerun report at all, so a report + # left behind by an earlier run would be read as belonging to this one. + rm -f "${WP_CLI_TEST_JUNIT_DIR}"/*.xml "${WP_CLI_TEST_JUNIT_DIR}"/rerun/*.xml + mkdir -p "${WP_CLI_TEST_JUNIT_DIR}/rerun" # The first --out belongs to the progress format and keeps it on stdout, the