Skip to content

Incremental rebuilds and see-also (5/6) - #8

Merged
tobiaskaestner merged 5 commits into
tkaestner/4-implementation-layerfrom
tkaestner/5-incremental-rebuilds
Oct 2, 2026
Merged

Incremental rebuilds and see-also (5/6)#8
tobiaskaestner merged 5 commits into
tkaestner/4-implementation-layerfrom
tkaestner/5-incremental-rebuilds

Conversation

@tobiaskaestner

@tobiaskaestner tobiaskaestner commented Oct 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Part 5 of 6 of a stacked series. It is based on #7 (tkaestner/4-implementation-layer); review the commits below only.

  • Re-read a document when its needs TOML changes. Before, an incremental build
    after a new link type crashed in sphinx-needs (KeyError: Link type ... does not exist in backlinks).
  • Re-read a document when the needs that it imports from a peer change.
    Before, a new incoming link did not show until a clean build.
  • A tag-file reference links into the document whose tag file defines it, not
    always into the API reference.
  • A @see item without a ref shows as a literal, not as nothing.
  • Read every @see section of a test. Doxygen writes one section per @see
    line.

Commits

  • 0935f15 fix: sphinx: re-read a document when its needs TOML changes
  • 04c57c9 fix: sphinx: re-read a document when the needs it imports change
  • 89d76fa fix: testmodule: link a tag-file reference into its own document
  • 9adbe8e fix: testmodule: render a see-also item without a ref as a literal
  • d42b423 fix: testmodule: read every see section of a test

Testing

  • Unit suite (python3 -m pytest sphinx/_extensions/_tests -q) at the head of this PR: 364 passed.
  • The safety docset (about 15 documents, 5 Doxygen projects) builds clean on the top of the stack, with all its gates.
  • The acceptance steps for these changes are in zdocs-tests (one PR after this series). With the engine at the top of the stack, that suite passes 321 of 321.

The stack

  1. Registry and build plumbing (1/6) #4 Registry and build plumbing
  2. Requirement links from Doxygen (2/6) #5 Requirement links from Doxygen
  3. Test report correctness (3/6) #6 Test report correctness
  4. Implementation layer and conditions (4/6) #7 Implementation layer and conditions
  5. Incremental rebuilds and see-also (5/6) #8 Incremental rebuilds and see-also (this PR)
  6. Coverage adequacy (6/6) #9 Coverage adequacy

🤖 Generated with Claude Code

This was referenced Oct 1, 2026
Merged
Merged
Merged
Merged
Merged
@tobiaskaestner
tobiaskaestner added this pull request to stack #10 October 1, 2026 18:47
@tobiaskaestner
tobiaskaestner force-pushed the tkaestner/5-incremental-rebuilds branch 2 times, most recently from 83207d3 to f8e5577 Compare October 2, 2026 15:05
tobiaskaestner and others added 5 commits October 2, 2026 17:07
@tobiaskaestner @claude
fix: sphinx: re-read a document when its needs TOML changes
e1442c5 
An incremental build after needs_config.toml gained a link type crashed
in sphinx-needs: KeyError: "Link type 'fulfills' does not exist in
backlinks." A clean build worked. sphinx-needs registers needs_types,
needs_links and needs_fields with rebuild "html", and needs_from_toml is
only the file's path, so a change to the file re-read no document. The
pickled needs had no backlink entry for the new link, and the first need
that used it (here an imported one, from a peer's needs.json) broke the
write phase. The removal of a link type that needs used crashed the same
way: the importing document's pickled environment still held a peer's
needs under the old vocabulary.

A new engine extension, needs_config_state, always loaded after
sphinx_needs, registers zdocs_needs_config_digest (rebuild "env") and
sets it at config-inited to the SHA-256 of the TOML file's content,
resolved against the conf dir as sphinx-needs resolves it. A change to
the content re-reads the document; an unchanged file, even rewritten,
keeps the cache, and both build stages see the same value. The extension
also records the digest an environment was built with
(zdocs-needs-config.sha256 in the doctree directory). At config-inited,
before Sphinx loads the environment, it deletes a pickled environment
built with another digest or with none, so the document starts fresh,
as in a clean build.

Reproduced on the safety docset with the extension removed: a probe link
type in needs_config.toml and one need using it; requirements-html then
failed with "Link type 'probes' does not exist in backlinks". With the
extension, doc-index + all-docs pass after the link type is added and
again after it is removed. Unit tests: the digest (unset, missing,
content not mtime, relative path), an incremental build that adds a link
type and uses it, removal by a peer with an importing document, the
stamp logic, an unchanged file keeping the cache, and a control: without
the extension the build raises the KeyError.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tobias Kaestner <tobias.kaestner@inovex.de>
@tobiaskaestner @claude
fix: sphinx: re-read a document when the needs it imports change
3da21bd 
An incremental build left "assessed by" off the ZEP-SRS-5-5 page of the
safety requirements. The requirements document imports the needs of the
test report (needs_external_needs). The imported needs.json gained
adequacy needs that link to the requirement, but the source of the
requirement page did not change. sphinx-needs loads the imports again on
each build, but Sphinx writes only the pages of the documents that it
read. So the page was not written again, and the new incoming link did not
show. aea18a5 and 36802e4 cover only a change of the needs TOML.

needs_config_state now also registers zdocs_imported_needs_digest
(rebuild "env"). At config-inited, it is set to a SHA-256 over the needs
that each json_path source of needs_external_needs gives: the needs of
the imported version and its schema, without the <link>_back fields.
sphinx-needs does not import those fields. When two documents import each
other, those fields would also make each document read again one more
time after each change. A change of the digest drops the pickled
environment, as a change of the TOML does, so the document starts from a
fresh one. This also removes the needs that a peer no longer has. The
stamp beside the environment now holds both digests, so each build dir
starts once from a fresh environment after this change.

Reproduced first. In a minimal project (peer and importer), a new peer
need that verifies an importer need did not show on the importer page
after an incremental build. On the safety docset (bdoc-p4-ef), a build
without ZDOCS_COVERAGE_OUT and then an incremental build with it left the
page without "assessed by", although requirements/needs.json had the
assesses_back entry. With the fix, the same sequence shows "assessed by".
In the next make with no change, the stage-1 builds of 4 documents that
import the test report started from a fresh environment once: they run
before the test report's stage 1, so they saw its new needs.json only
then. The make after that changed no digest.

Unit tests: the digest (no source, json_url only, the imported needs,
_back fields and creator ignored, the configured version, a relative
path), an incremental build where a peer gains a need and one where it
removes one, an unchanged import that keeps the cache, and a control:
without the extension, the new link does not show.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tobias Kaestner <tobias.kaestner@inovex.de>
@tobiaskaestner @claude
fix: testmodule: link a tag-file reference into its own document
b4755fd 
A symbol Doxygen resolved through a tag file carries the tag file's path
in external=. ref_to_rst sent every such reference to the api_reference
document (doxygen_parser.py, `links.api if ref.get("external")`). Once
the safety docset's Detailed Design documented kernel internals, the
cpu_mask test's mention of cpu_mask_mod() resolved through the Detailed
Design's tag file and rendered as a dead link into the Safety API
(dox-zephyr-safety-api/cpu__mask_8c.html), accepted in doc-check since.

docrefs.tag_urls maps each tag file the registry can name to its
document's HTML directory: a kind: doxygen peer relative to this
document's root, a doxygen-external peer by its remote directory, a needs
tag by its Sphinx root. The keys are the paths tagfiles() hands Doxygen,
which is what external= repeats. The testmodule block carries the map,
zdocs_conf sets it as testmodule_tag_urls, and the testmodule directive
prefixes the relative ones for the page as it does api_doxygen_url.
RefLinks gains `tags`; ref_to_rst looks the reference's tag file up
(paths compared normalised) and falls back to `api` for a tag file the
map does not name, which is also what a caller without a map gets.
parse_memberdef, see_to_rst and build_procedure_need_rst pass it on.

On the safety docset the link now goes to
dox-zephyr-safety-detailed-design/cpu__mask_8c.html#ac8b9..., and
doc-check is OK without the accepted finding. Unit tests: routing by tag
file in prose, see-also and a whole memberdef, normalised paths, the
fallbacks, the page prefix, and docrefs' map, including that its keys are
the paths tagfiles() writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tobias Kaestner <tobias.kaestner@inovex.de>
@tobiaskaestner @claude
fix: testmodule: render a see-also item without a ref as a literal
3c48faa 
see_to_rst() read only the <ref> elements of a see section. When no
Doxygen project documents a symbol, `@see irq_offload()` gives no <ref>,
only the text <para>irq_offload()</para>. So the see section gave no
"See also" line, and the reference was lost. On the safety docset, the
"See also" line of TSPEC-COMMON-068, -069, -075 and TSPEC-LOGGING-008
was empty. In a section with refs, a name without a ref was lost too, for
example log_stack_usage() after k_thread_foreach().

see_to_rst() now reads each <para> of the section in the order of the
source. A <ref> gives the same RST as before, also inside a
<computeroutput>. The text between the refs is divided into items at the
commas that are not in parentheses, and at the space after a ")". Each
item becomes a literal. A piece with no letter or digit, for example the
", " between two refs or a final ".", gives no item. So a line that has
only refs is the same as before.

Measured on the testspec XML of the safety docset, for the 912 needs of
the test specification with a see section: 617 give the same line, 287
that gave no line give one now, and 8 get more items. In these 8, the
links do not change and stay in the same order. The 4 needs above now
show "See also: irq_offload()" or "See also: printk()".

Unit tests: text without a ref, refs and text in order, separators, a
comma in parentheses, a space after a call, a ref in a computeroutput, a
section with only separators.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tobias Kaestner <tobias.kaestner@inovex.de>
@tobiaskaestner @claude
fix: testmodule: read every see section of a test
b369988 
parse_memberdef() and build_procedure_need_rst() read only the first
see section of a member (dd.find). Doxygen 1.16 writes one see section
for each `@see` line, also for lines that follow each other. So a test
with `@see k_thread_join()` and then `@see irq_offload()` showed only
k_thread_join(). On safety main, 5 references to a suite group were
hidden this way: irq_offload() in TSPEC-MEMPROT-019 and TSPEC-THREADS-030,
-049, -050 and -053.

Both callers now give all see sections (dd.findall) to see_to_rst(),
which takes one section or a list of them. The line has the items of all
sections, in the order of the source.

Measured on the testspec XML of the safety docset: 353 needs of the test
specification have more than one see section, and all 353 get more
items. No need gets an item two times.

Unit tests: two sections joined in order, an empty list, parse_memberdef
with two sections (the TSPEC-THREADS-030 shape), and a procedure with
two sections.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tobias Kaestner <tobias.kaestner@inovex.de>
@tobiaskaestner
tobiaskaestner force-pushed the tkaestner/5-incremental-rebuilds branch from f8e5577 to b369988 Compare October 2, 2026 15:07
@tobiaskaestner
tobiaskaestner merged commit b5238fa into main Oct 2, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@tobiaskaestner