Test report correctness (3/6)

doc/manual/explanation/testmodule-and-twister.rst

@@ -71,6 +71,34 @@ are **derived from the registry** — the report's own ``testmodule:`` block
 already names the specification it reads — rather than hand-written by the
 consumer, which would duplicate what the registry knows.
 
+Parameterized tests
+-------------------
+
+A ztest ``ZTEST_P`` function runs once per parameter value, and twister
+reports it twice over: one aggregate result, ``<scenario>.<suite>.<fn>``, from
+ztest's summary line, and one result per value,
+``<scenario>.<fn>[<instantiation>/<value>]`` — with no suite segment. The
+specification documents the function once, so the report does too: each value
+result is attached to the aggregate of the same run (platform and scenario),
+and the aggregate's result need carries them. Its status comes from the
+values — failed if any failed, skipped if all were skipped, passed if every
+value that ran passed — and its body counts them ("9 values: 8 passed, 1
+failed") and tabulates only the values that did not pass, with the assertion
+each one failed on.
+
+The aggregate's own status is not trusted for this. When one value fails,
+ztest summarises the function as ``FLAKY``, which twister does not recognise:
+it records the aggregate as ``blocked`` in ``twister.json`` and as a generic
+"Testsuite failed" in the XML, and only the failing value's result carries the
+real assertion. Where twister's status for the aggregate disagrees with the
+values' verdict, the need says so ("Twister reported the test as
+``blocked``").
+
+A run whose values have no aggregate at all borrows the suite from other runs'
+aggregates of the same scenario and function, or else from the specification
+if exactly one test case has that function name. Values that match neither are
+skipped with one warning per function, not one per value.
+
 Where the test runner writes
 ----------------------------
 

doc/manual/howto/render-test-specifications.rst

@@ -105,6 +105,12 @@ for the scenario table.
 
    .. twisterinfo:: twister.json
 
+``:module:`` selects the runs by scenario-name prefix. Where one module's
+scenario name is a prefix of another's (upstream Zephyr has many), select by
+test directory instead: ``:path:`` takes the testsuite path as ``twister.json``
+records it, relative to ``ZEPHYR_BASE`` — for a test root outside the Zephyr
+tree that starts with ``../`` (see :doc:`../reference/directives-and-roles`).
+
 Point ``ZDOCS_TWISTER_OUT`` at a real ``west twister`` output directory
 (``-DZDOCS_TWISTER_OUT=$(west topdir)/twister-out``). Leaving it unset is
 supported: both directives render a "not found" note and the build still

doc/manual/reference/directives-and-roles.rst

@@ -106,15 +106,46 @@ groups becomes one need each — nothing is written by hand per test case.
 .. code-block:: rst
 
    .. testreport:: twister_report.xml
-      :module: widget.probe
+      :path: tests/kernel/timer/timer_error_case
 
    .. twisterinfo:: twister.json
 
 ``testreport``'s and ``twisterinfo``'s arguments are filenames resolved
 against ``ZDOCS_TWISTER_OUT`` (or the including document's own directory, as a
 fallback, if that is unset) unless given as an absolute path.
-``testreport``'s optional ``:module:`` prefix-matches against the JUnit
-``classname``. Both directives **soft-fail** to a short "not found" paragraph
+
+``testreport`` selects which runs a page shows with two optional options:
+
+``:path:``
+   A test directory, exactly as twister records it in ``twister.json``'s
+   testsuite ``path`` — relative to ``ZEPHYR_BASE``, e.g.
+   ``tests/kernel/timer/timer_error_case`` (a test root outside the Zephyr
+   tree reads ``../<project>/tests/...``). Compared exactly after normalising
+   slashes, a leading ``./`` and a trailing ``/``; never as a prefix. The path
+   comes from the ``twister.json`` beside the report XML (the XML has none), and
+   each result is matched to its testsuite by platform and scenario name. If
+   that ``twister.json`` is missing, the directive soft-fails to a "not found"
+   paragraph like the other inputs.
+``:module:``
+   A scenario-name prefix, matched against the JUnit ``classname`` (the
+   scenario itself, or ``<module>.`` followed by anything).
+
+With both, a run must match both. With neither, the page shows every result in
+the report. ``:path:`` is the one that identifies a test module: scenario
+names do not follow directories upstream — tests/kernel/timer/timer_api runs
+as ``kernel.timer``, a prefix of timer_error_case's ``kernel.timer.error_case``
+— so ``:module:`` alone can put one module's results on another's page, where
+the second page then fails with "A need with ID … already exists". The
+execution-log section and the result summary follow the same selection, so a
+page is consistent with itself.
+
+A parameterized test (``ZTEST_P``) gets one result need per run, not one per
+parameter value: the values' results are attached to the test's aggregate
+result, which takes its status from them and lists the values that did not
+pass (:doc:`../explanation/testmodule-and-twister`). No need type or field is
+added for this; the values render in the need's body.
+
+Both directives **soft-fail** to a short "not found" paragraph
 when their input is absent, rather than failing the build — a documentation
 build outrunning its test run is a normal pipeline state. ``testmodule`` does
 **not** soft-fail on a missing Doxygen group: annotated source is expected to

sphinx/_extensions/_tests/fixtures/twister-param/needs.json

@@ -0,0 +1,29 @@
+{
+  "current_version": "1.0",
+  "versions": {
+    "1.0": {
+      "needs": {
+        "TSPEC-SEM-001": {
+          "id": "TSPEC-SEM-001",
+          "type": "test_case",
+          "title": "sem init validity",
+          "test_function": "test_sem_init_validity",
+          "test_module": "tests/kernel/semaphore/semaphore",
+          "suite": "semaphore",
+          "suite_title": "Semaphore",
+          "verifies": []
+        },
+        "TSPEC-SEM-002": {
+          "id": "TSPEC-SEM-002",
+          "type": "test_case",
+          "title": "sem count get",
+          "test_function": "test_sem_count_get",
+          "test_module": "tests/kernel/semaphore/semaphore",
+          "suite": "semaphore",
+          "suite_title": "Semaphore",
+          "verifies": []
+        }
+      }
+    }
+  }
+}

sphinx/_extensions/_tests/fixtures/twister-param/twister.json

@@ -0,0 +1,77 @@
+{
+  "environment": {
+    "run_date": "2026-09-29T12:18:23+00:00",
+    "zephyr_version": "v4.4.0-13457-g24e1b45a5d81",
+    "toolchain": "zephyr/gnu",
+    "os": "Linux"
+  },
+  "testsuites": [
+    {
+      "name": "kernel.semaphore",
+      "arch": "arm",
+      "platform": "mps2/an385",
+      "toolchain": "zephyr/gnu",
+      "status": "failed",
+      "reason": "Testsuite failed",
+      "path": "tests/kernel/semaphore/semaphore",
+      "testcases": [
+        {
+          "identifier": "kernel.semaphore.semaphore.sem_count_get",
+          "execution_time": "0.01",
+          "status": "passed"
+        },
+        {
+          "identifier": "kernel.semaphore.semaphore.sem_init_validity",
+          "execution_time": "0.00",
+          "status": "blocked",
+          "reason": "Testsuite failed"
+        },
+        {
+          "identifier": "kernel.semaphore.sem_init_validity[cases/0]",
+          "execution_time": "0.00",
+          "status": "passed"
+        },
+        {
+          "identifier": "kernel.semaphore.sem_init_validity[cases/1]",
+          "execution_time": "0.00",
+          "status": "passed"
+        },
+        {
+          "identifier": "kernel.semaphore.sem_init_validity[cases/2]",
+          "execution_time": "0.00",
+          "status": "passed"
+        },
+        {
+          "identifier": "kernel.semaphore.sem_init_validity[cases/3]",
+          "execution_time": "0.00",
+          "status": "passed"
+        },
+        {
+          "identifier": "kernel.semaphore.sem_init_validity[cases/4]",
+          "execution_time": "0.00",
+          "status": "passed"
+        },
+        {
+          "identifier": "kernel.semaphore.sem_init_validity[cases/5]",
+          "execution_time": "0.00",
+          "status": "passed"
+        },
+        {
+          "identifier": "kernel.semaphore.sem_init_validity[cases/6]",
+          "execution_time": "0.00",
+          "status": "passed"
+        },
+        {
+          "identifier": "kernel.semaphore.sem_init_validity[cases/7]",
+          "execution_time": "0.00",
+          "status": "passed"
+        },
+        {
+          "identifier": "kernel.semaphore.sem_init_validity[cases/8]",
+          "execution_time": "0.01",
+          "status": "failed"
+        }
+      ]
+    }
+  ]
+}

sphinx/_extensions/_tests/fixtures/twister-param/twister_report.xml

@@ -0,0 +1,34 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- Extracted from a real twister run (Zephyr v4.4.0, mps2/an385) of tests/kernel/semaphore/semaphore
+     with value 8 of sem_init_validity's ZTEST_P made to fail. ztest summarised the test as
+     "FLAKY - (Failed 1 of 9 attempts)", which twister does not recognise: the aggregate gets
+     "Testsuite failed" here and `blocked` in twister.json. Only [cases/8] carries the assertion.
+     The aggregate's failure text (the whole handler log) is shortened. -->
+<testsuites>
+	<testsuite name="mps2/an385" tests="11" failures="2" errors="0" skipped="0">
+		<testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore.sem_count_get" time="0.01" />
+		<testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore.sem_init_validity" time="0.00">
+			<failure type="failure" message="Testsuite failed">*** Booting Zephyr OS build v4.4.0-13457-g24e1b45a5d81 ***
+Running TESTSUITE semaphore
+[...]
+TESTSUITE semaphore failed.
+</failure>
+		</testcase>
+		<testcase classname="kernel.semaphore" name="kernel.semaphore.sem_init_validity[cases/0]" time="0.00" />
+		<testcase classname="kernel.semaphore" name="kernel.semaphore.sem_init_validity[cases/1]" time="0.00" />
+		<testcase classname="kernel.semaphore" name="kernel.semaphore.sem_init_validity[cases/2]" time="0.00" />
+		<testcase classname="kernel.semaphore" name="kernel.semaphore.sem_init_validity[cases/3]" time="0.00" />
+		<testcase classname="kernel.semaphore" name="kernel.semaphore.sem_init_validity[cases/4]" time="0.00" />
+		<testcase classname="kernel.semaphore" name="kernel.semaphore.sem_init_validity[cases/5]" time="0.00" />
+		<testcase classname="kernel.semaphore" name="kernel.semaphore.sem_init_validity[cases/6]" time="0.00" />
+		<testcase classname="kernel.semaphore" name="kernel.semaphore.sem_init_validity[cases/7]" time="0.00" />
+		<testcase classname="kernel.semaphore" name="kernel.semaphore.sem_init_validity[cases/8]" time="0.01">
+			<failure type="failure" message="Testsuite failed">START - test_sem_init_validity[cases/8]
+
+    Assertion failed at CMAKE_SOURCE_DIR/src/main.c:363: semaphore_test_sem_init_validity: (_act not equal to _exp)
+k_sem_init incorrect return value: -22 != 0
+ FAIL - test_sem_init_validity[cases/8] in 0.006 seconds
+</failure>
+		</testcase>
+	</testsuite>
+</testsuites>

sphinx/_extensions/_tests/fixtures/twister-path/needs.json

@@ -0,0 +1,21 @@
+{
+  "current_version": "1.0",
+  "versions": {
+    "1.0": {
+      "needs": {
+        "TSPEC-TA-001": {
+          "id": "TSPEC-TA-001", "type": "test_case", "title": "timer duration period",
+          "test_function": "test_timer_duration_period",
+          "test_module": "tests/kernel/timer/timer_api",
+          "suite": "timer_api", "suite_title": "Timer API", "verifies": []
+        },
+        "TSPEC-TE-001": {
+          "id": "TSPEC-TE-001", "type": "test_case", "title": "timer start null",
+          "test_function": "test_timer_start_null",
+          "test_module": "tests/kernel/timer/timer_error_case",
+          "suite": "timer_api_error", "suite_title": "Timer error cases", "verifies": []
+        }
+      }
+    }
+  }
+}

sphinx/_extensions/_tests/fixtures/twister-path/twister.json

@@ -0,0 +1,30 @@
+{
+  "environment": {
+    "run_date": "2026-09-29T10:00:00+00:00",
+    "zephyr_version": "4.4.0",
+    "toolchain": "zephyr",
+    "os": "linux"
+  },
+  "testsuites": [
+    {
+      "name": "kernel.timer",
+      "platform": "mps2/an385",
+      "path": "tests/kernel/timer/timer_api",
+      "toolchain": "zephyr/gnu",
+      "status": "passed",
+      "testcases": [
+        {"identifier": "kernel.timer.timer_api.timer_duration_period", "status": "passed"}
+      ]
+    },
+    {
+      "name": "kernel.timer.error_case",
+      "platform": "mps2/an385",
+      "path": "tests/kernel/timer/timer_error_case",
+      "toolchain": "zephyr/gnu",
+      "status": "passed",
+      "testcases": [
+        {"identifier": "kernel.timer.error_case.timer_api_error.timer_start_null", "status": "passed"}
+      ]
+    }
+  ]
+}

sphinx/_extensions/_tests/fixtures/twister-path/twister_report.xml

@@ -0,0 +1,10 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- Two test modules whose scenario names prefix-collide, as upstream's do:
+     tests/kernel/timer/timer_api runs as `kernel.timer`, and
+     tests/kernel/timer/timer_error_case as `kernel.timer.error_case`. -->
+<testsuites>
+  <testsuite name="mps2/an385" tests="2" failures="0" errors="0" skipped="0">
+    <testcase classname="kernel.timer" name="kernel.timer.timer_api.timer_duration_period" time="0.10" />
+    <testcase classname="kernel.timer.error_case" name="kernel.timer.error_case.timer_api_error.timer_start_null" time="0.20" />
+  </testsuite>
+</testsuites>

sphinx/_extensions/_tests/roots/test-testreport-param/conf.py

@@ -0,0 +1,50 @@
+# Copyright (c) 2026 inovex GmbH
+#
+# SPDX-License-Identifier: Apache-2.0
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[3]))  # _extensions/
+
+_FIXTURES = Path(__file__).resolve().parents[2] / "fixtures"
+
+extensions = ["sphinx_needs", "test_module"]
+master_doc = "index"
+exclude_patterns = ["_build"]
+
+needs_types = [
+    dict(directive="test_result", title="Test Result", prefix="TRESULT_",
+         color="#FCE4D6", style="node"),
+    dict(directive="test_case",   title="Test Case",   prefix="TCASE_",
+         color="#E2EFDA", style="node"),
+]
+_str_field = {"schema": {"type": "string"}, "nullable": True}
+needs_fields = {
+    "platform":       {**_str_field},
+    "scenario":       {**_str_field},
+    "twister_id":     {**_str_field},
+    "execution_time": {**_str_field},
+    "reason":         {**_str_field},
+    "test_function":  {**_str_field},
+    "test_module":    {**_str_field},
+    "suite":          {**_str_field},
+    "suite_title":    {**_str_field},
+}
+needs_id_regex = r"^[A-Za-z][A-Za-z0-9_-]+"
+needs_links = {
+    "result_of": {"description": "result of",  "incoming": "has results",  "outgoing": "result of"},
+    "covers":    {"description": "covers",      "incoming": "covered by",   "outgoing": "covers"},
+    "verifies":  {"description": "verifies",    "incoming": "verified by",  "outgoing": "verifies"},
+}
+needs_external_needs = [{
+    "json_path": str(_FIXTURES / "twister-param" / "needs.json"),
+    "base_url": "http://localhost/",
+    "version": "1.0",
+}]
+twister_output_dir = str(_FIXTURES / "twister-param")
+testspec_needs_json = str(_FIXTURES / "twister-param" / "needs.json")
+testspec_doxygen_url = "testspec"
+api_doxygen_url = "api"
+needs_build_json = True
+suppress_warnings = ["needs.link_outgoing", "needs.external_link_outgoing", "config.cache"]

sphinx/_extensions/_tests/roots/test-testreport-param/index.rst

@@ -0,0 +1,5 @@
+Test Report
+===========
+
+.. testreport:: twister_report.xml
+   :path: tests/kernel/semaphore/semaphore