Test report correctness (3/6) - #6
Merged
Merged
Conversation
This was referenced Oct 1, 2026
Merged
tobiaskaestner
added this pull request to stack #10
October 1, 2026 18:47
tobiaskaestner
force-pushed
the
tkaestner/3-test-report-correctness
branch
2 times, most recently
from
October 2, 2026 15:05
8895412 to
9a9fcde
Compare
The test report found the spec test case for a twister result by function name alone. ZTEST function names are not unique across suites -- some 300 are reused in the Zephyr test tree -- so a result silently attached to whichever test case of that name was read last. load_spec_lookup now returns a SpecLookup keyed by (suite, test_function). It tries the name as reported and with ztest's test_ prefix, and falls back to the bare name only when exactly one test case carries it. The four inline lookups in test_module.py become one helper, which warns and links nothing when a result has no match or several, naming the candidates. That also covers one (suite, function) pair documented in several test modules, which the Zephyr tree has 26 of; resolving those needs the test module and is left for later. Five new tests drive load_spec_lookup and _build_results_rst together. Against the unchanged code 4 fail; the fifth pins the unique-name fallback, which already worked. Suite 158 -> 164. The safety test report is unchanged at queue/fifo scope: 90 results, same links. The acceptance suite (zdocs-tests) was not available and has not been run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tobias Kaestner <tobias.kaestner@inovex.de>
testreport reads the twister XML and the spec's needs.json, and twisterinfo reads twister.json. All of them are outside the Sphinx source tree, and none were noted as dependencies. After a new twister run into the same ZDOCS_TWISTER_OUT, an incremental build kept the old report, or the "not found" one, and published it without a warning. Reproduced in safety-toolbox: 0 results before `west twister`, still 0 after. A Sphinx dependency alone is not enough. Sphinx re-reads a document only when a dependency is missing or NEWER than the document's last read. Twister output often arrives OLDER than that: a CI artifact, a cache restored with its timestamps, or a copy that keeps them. zdocs-tests step 31 reproduces it with shutil.copy2. Note each input with env.note_dependency, before the existence checks, so a report built ahead of its test run is re-read until the output appears. Also record each input's signature (mtime_ns, size, or None when missing) per document while it is read. An env-get-outdated handler re-reads any document whose signature differs. env-purge-doc and env-merge-info keep the record per document under parallel reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tobias Kaestner <tobias.kaestner@inovex.de>
Two defects in `.. testmodule::`, both of which kept a stale test specification after the test sources changed. 1. None of its inputs were tracked. It reads Doxygen's index.xml, the module group, every inner group, and testcase.yaml, all outside the Sphinx source tree. An incremental build after a test change (a retagged status, a renamed test) therefore kept the old test cases. Reproduced in safety-toolbox: two tests retagged @draft/@obsolete, a fresh build shows 17/1/1 active/draft/obsolete, an incremental build kept 19 active. Now each input is noted through _note_input (4da0074), so any signature change re-reads the page. 2. The group index was cached as env._testmodule_group_index. env is pickled across builds, so a module group added or renamed later stayed "not found" until a full rebuild. It is now cached on the app, which lives for one build. testreport, twisterinfo and testmodule now all record their inputs through the one _note_input helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tobias Kaestner <tobias.kaestner@inovex.de>
testreport selected results with :module:, a scenario-name prefix. Upstream scenario names do not follow module directories: tests/kernel/timer/timer_api runs as `kernel.timer`, which prefix-matches timer_error_case's `kernel.timer.error_case`, and tests/kernel/fatal/exception runs as `kernel.common.stack_protection*`. In the safety build 1431 of 3630 results rendered on another module's page; whichever page was built first won, and the others produced 3960 "A need with ID 'TR-...' already exists" warnings. The new :path: option names the test directory as twister.json records it (relative to ZEPHYR_BASE) and matches it exactly, after normalising slashes, a leading ./ and a trailing /. twister_report.xml has no path, so results are matched to twister.json's testsuites by (platform, scenario). The twister.json is the one beside the report XML; if it is missing the directive soft-fails to a "not found" paragraph like its other inputs, and it is tracked as an input so the page is re-read once it appears. :module: works as before. Given both, a run must match both. The execution logs use the same selection, so a page never shows a log whose results it does not show; the summary table already follows the selected results. On twister-out-b3, :path: tests/kernel/timer/timer_api selects 412 results (kernel.timer, kernel.timer.no_multitheading) where :module: kernel.timer took 508 from eight scenarios. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tobias Kaestner <tobias.kaestner@inovex.de>
A ztest ZTEST_P function runs once per parameter value. Twister reports
one aggregate <scenario>.<suite>.<fn> from ztest's summary plus one result
per value, <scenario>.<fn>[<instantiation>/<value>], with no suite segment.
The report parsed a value as suite "" and function "sem_init_validity[cases/0]",
found no spec case, and skipped it with a warning: 618 warnings in the
safety build, and the per-value results were lost.
Those are the results that matter when a value fails. ztest then summarises
the function as FLAKY, which twister's summary parser does not know, so the
aggregate becomes `blocked` in twister.json and a generic "Testsuite failed"
in the XML. Only the failing value's result carries the assertion.
Values are now recognised and attached to the aggregate of the same run
(platform + scenario, matched by function name; the suite comes from the
aggregate). The spec keeps one test case, and the report one result need
per run. That need takes its status from the values (failed if any failed,
skipped if all skipped, else passed), shows twister's own status when it
disagrees ("Twister reported the test as `blocked`"), and renders "9 values:
8 passed, 1 failed" plus a table of the values that did not pass, with the
assertion text instead of the suite message. A run without an aggregate
takes the suite from other runs' aggregates of the same scenario and
function, or from a unique spec case; values that match neither are
skipped with one warning per function. twister.json is read, if present,
for the statuses the XML cannot express.
The fixture is cut from a real run with value 8 of sem_init_validity made
to fail. On twister-out-b3 the 467 value results fold into their 21
aggregates, leaving the 3630 results the report shows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tobias Kaestner <tobias.kaestner@inovex.de>
tobiaskaestner
force-pushed
the
tkaestner/3-test-report-correctness
branch
from
October 2, 2026 15:07
9a9fcde to
d7fb9fe
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 3 of 6 of a stacked series. It is based on #5 (
tkaestner/2-doxygen-requirement-links); review the commits below only.names repeat across suites, so a join by function alone gives wrong results.
than the last build (a CI artifact or a restored cache).
testmoduletracks its inputs, and the group index is no longer pickled.testreport :path:selects results by the test directory intwister.json.A scenario prefix also picked up the results of other modules.
fn[inst/N]) go to the one result ofthat test.
Commits
Testing
python3 -m pytest sphinx/_extensions/_tests -q) at the head of this PR: 229 passed.The stack
🤖 Generated with Claude Code