Skip to content

Both reference tables in the README, generated from the package's own docstrings - #6

Closed
borisdev wants to merge 1 commit into
coherence-findingfrom
well-formedness-rules
Closed

borisdev wants to merge 1 commit into
coherence-findingfrom
well-formedness-rules

Conversation

@borisdev

@borisdev borisdev commented Oct 1, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #5 — base is coherence-finding, because these tables name coherence_check() and that branch is where the rename lands. Merge #5 first.

The README said what the library is for and never what it checks. A reader could not tell whether coherence_check() was three trivial rules or eleven real ones — and the data language, the whole reason the design is inspectable at all, was never written down anywhere.

Two generated tables, in the README

the data language every word you declare a design with, grouped into values / boxes / wires / endpoints / the design
well-formedness every rule coherence_check() enforces, split by whether it needs a strategy
python3 -m workflow_workbench.reference --write
python3 -m workflow_workbench.reference --check

No description is authored. Every cell is the first line of the thing's own docstring — or, for the two unions, which have no docstring of their own, the names of their members. A table cannot claim what the code does not say.

What is authored is the grouping: which word sits under "boxes" and which under "wires". That's curation, it's source, and there's a test asserting GROUPS carries names only — so the table can never become half source and half derived.

The README opening now names the failure, not the audience

Your framing: an agent left to run free produces work that runs and is incoherent — each piece locally fine, the whole not adding up. That's a reason to care. "For developing workflows with an AI coding agent" only said who it was for.

Four problems, one declaration answering all four: incoherence, strategies that can only be compared rather than asserted, reuse, and visibility.

⚠️ The visualization claim is narrowed to what's true

pydantic-graph does emit mermaid — build_mermaid_graph in graph_builder.py — so "they have none" would have been false. Measured: it isn't exported from their __init__, and it takes a built graph's internals, so every implementation must exist first. Two honest differences, not a feature list: ours reads the declaration, and ours can draw two strategies at once.

Fixed at source, not worked around

  • _Start's first docstring line ended mid-sentence on "so mypy can narrow a"; _End's was "See _Start." Both are standalone sentences now, because the table lifts that line verbatim. A test makes a wrapping summary a failure rather than a fragment in the README.
  • The union rows needed their pipes escaped — a bare | is a column separator, so StepSpec | JoinSpec | DecisionSpec rendered as three empty cells. Invisible in the source string; the test asserts the rendered row.
  • The generated prose said NodeSpec(...) is a TypeError, which test_the_docs_use_the_current_api greps for. Reworded at source rather than exempting the README — that lint exists because the README's own example once stopped running and nothing said so, and the README is the last file that should be exempt. It caught me twice in this change, both times correctly.

docs/well-formedness.md is deleted — the tables live in the README now, and two homes for one generated table is exactly the duplication this repo keeps drift tests to avoid.

Mutation-tested

Six ways this can rot, introduced one at a time, each went red:

mutation caught by
hand-edit a README table row the --check drift test
a docstring changes, nobody regenerates same
a new export left out of the table the completeness guard
a stale exclusion naming something no longer exported the exclusion-existence test
GROUPS grows a hand-written description the source/derived test
the unions lose pipe escaping the rendered-row test

245 tests pass. parity --check and reference --check both green.

🤖 Generated with Claude Code

Copilot AI balanced review requested due to automatic review settings October 1, 2026 16:15

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The inventory omits an enforced recursion rule, and the documentation describes a nonexistent structured finding API.

Review effort: Balanced
Findings: 1 Medium severity · 1 Low severity

Open (2)
What changed in this PR

Adds generated documentation for the well-formedness checks enforced by check().

Changes:

  • Adds a rules documentation generator and drift checks.
  • Adds generated README and detailed documentation sections.
  • Adds tests for completeness, grouping, and generated counts.
File Description
workflow_workbench/​rules.py Generates rule documentation from check metadata.
tests/​test_rules.py Tests generation and rule discovery.
README.md Adds a generated rules summary.
docs/​well-formedness.md Documents checks and findings.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread workflow_workbench/rules.py Outdated
def rules() -> tuple[Rule, ...]:
"""Every exported check, in the order `checks.__all__` declares them."""
out = []
for name in checks.__all__:
Comment thread docs/well-formedness.md Outdated
Comment on lines +53 to +62
`check()` returns `CoherenceFinding`, a `str` subclass. The sentence is the finding; the fields
are there so a caller — or a coding agent using `check()` as its acceptance test — can branch
without parsing English.

```python
f = spec.check(strategy)[0]
f.check # which rule above produced it
f.about # the node, 'source->target' for an edge, or '' for the whole design
f.blocking # False only for a `NOT CHECKED — …` stated gap
```
… docstrings

The README said what the library is FOR and never what it checks. A reader could not
tell whether `coherence_check()` was three trivial rules or eleven real ones, and the
data language — the whole reason the design is inspectable — was never written down.

TWO GENERATED TABLES, in the README where they were asked for:

    the data language     every word you declare a design with, grouped
    well-formedness       every rule coherence_check() enforces, split by whether it
                          needs a strategy

    python3 -m workflow_workbench.reference --write
    python3 -m workflow_workbench.reference --check

NO DESCRIPTION IS AUTHORED. Every cell is the first line of the thing's own docstring,
or — for the two unions, which have none of their own — the names of their members.
A table cannot claim what the code does not say.

What IS authored is the GROUPING: which word sits under "boxes" and which under
"wires". That is curation, it is source, and it is a literal in `reference.py`. The
source/derived split is the point, so there is a test asserting GROUPS carries names
only.

THE README OPENING NOW NAMES THE FAILURE, NOT THE AUDIENCE. Boris's framing: an agent
left to run free produces work that RUNS AND IS INCOHERENT — each piece locally fine,
the whole not adding up. That is a reason to care; "for developing workflows with an
AI coding agent" was only a statement of who it is for. Four problems, one declaration
answering all four: incoherence, strategies that can only be compared and not asserted,
reuse, and visibility.

⚠️ The visualization claim is narrowed to what is true. pydantic-graph DOES emit
mermaid — `build_mermaid_graph` in `graph_builder.py` — so "they have none" would have
been false. Measured: it is not exported from their `__init__`, and it takes a BUILT
graph's internals, so every implementation must exist first. Two honest differences
then, not a feature list: ours reads the declaration, and ours can draw two strategies
at once.

Rebased onto the coherence-finding branch, because these tables name the method and
that branch renames it. #6 is stacked on #5.

ALSO FIXED AT SOURCE, not worked around:

  · `_Start`'s first docstring line ended mid-sentence on "so `mypy` can narrow a", and
    `_End`'s was "See `_Start`." Both are now standalone sentences, because the table
    lifts that line verbatim. `test_every_first_docstring_line_stands_alone` makes a
    wrapping summary a test failure rather than a fragment in the README.
  · the union rows needed their pipes ESCAPED — a bare `|` is a column separator, so
    `StepSpec | JoinSpec | DecisionSpec` rendered as three empty cells. Invisible in
    the source string; the test asserts the rendered row.
  · the generated prose said "`NodeSpec(...)` is a `TypeError`", which
    `test_the_docs_use_the_current_api` greps for and flags. Reworded at the source
    rather than exempting the README — that lint exists because the README's own
    example once stopped running and nothing said so, and the README is the last file
    that should be exempt from it. It has now caught me twice in this change, both
    times correctly.

`docs/well-formedness.md` is deleted: the tables live in the README now, and two homes
for one generated table is the duplication this repo keeps a drift test to avoid.

Six ways this can rot were introduced one at a time, and each went red: hand-editing a
README row, a docstring changing without regeneration, a new export left out of the
table, a stale exclusion naming something no longer exported, GROUPS growing a
hand-written description, and the unions losing their pipe escaping.

245 tests pass. `parity --check` and `reference --check` both green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@borisdev
borisdev force-pushed the well-formedness-rules branch from 07bd73b to d7d8abf Compare October 1, 2026 16:39
Copilot AI balanced review requested due to automatic review settings October 1, 2026 16:39
@borisdev borisdev changed the title The well-formedness rules, generated from the checks themselves Both reference tables in the README, generated from the package's own docstrings Oct 1, 2026
@borisdev
borisdev changed the base branch from main to coherence-finding October 1, 2026 16:39

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The rules table omits recursive-subgraph cycle detection enforced by coherence_check().

Review effort: Balanced
Findings: 2 Medium severity · 1 Low severity

Open (3)

Comment on lines +148 to +150
for name in checks.__all__:
if not name.startswith("check_"):
continue
borisdev added a commit that referenced this pull request Oct 6, 2026
…— and five rounds of review (#5 #6 #7 #8)

Contains #5, #6 and #7 — the stack is linear, so this one merge brings all four.

- #5 `coherence_check()` returns `CoherenceFinding`, so an agent can branch instead of regex
- #6 both reference tables in the README, generated from the package's own docstrings
- #7 the glossary, linked from the prose, with a test that kills dead links
- #8 the thesis, `examples/contestable.py`, `StepSpec.problem`, the implement-a-workflow skill

Merged as a MERGE COMMIT, not a squash, so the eight commits stay in history.

## Five rounds of Copilot review, worked

20 inline comments were on record unread when consolidation started. Across five rounds:
14 live / 2 stale on round one, 5 on round two (three of them my own regression), pickle on
round three, a stale docstring on round four, and two previously-missed on round five.

⚠️ **Every single finding was a CHECK narrower than its claim, not code that was wrong.** The
dead-link regex skipped malformed links; the "every rule" table enumerated 11 of 12; the retired
-token map had no `.check(`; the compatibility oracle's five string operations omitted pickle;
the rule-count guard scanned one phrasing in one file; a UI test drove one of three render
surfaces. The code was mostly right. The things asserting it was right were the defects.

266 -> 278 tests. Both generators green. Bundle reproduces byte-for-byte from `npm run build`.

## One finding deliberately open

Nested findings propagate an `about` relative to the child design, so it is not resolvable by a
caller holding only the parent. Both readings are right about different things, and reconciling
them means `about` carrying a path across a boundary — a semantic change to the field. Tracked in
#9 with the limit documented in-code at both sites rather than left overstating.
@borisdev

borisdev commented Oct 6, 2026

Copy link
Copy Markdown
Owner Author

Contained in #8, merged as a merge commit — all eight commits are in main's history, so this branch's review record stands and nothing is lost. Closing rather than merging to avoid three successive squash-divergence resolutions.

@borisdev borisdev closed this Oct 6, 2026
@borisdev
borisdev deleted the well-formedness-rules branch October 6, 2026 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants