Repository navigation
Conversation
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The inventory omits an enforced recursion rule, and the documentation describes a nonexistent structured finding API.
Review effort: Balanced
Findings: 1
Open (2)
What changed in this PR
Adds generated documentation for the well-formedness checks enforced by check().
Changes:
- Adds a rules documentation generator and drift checks.
- Adds generated README and detailed documentation sections.
- Adds tests for completeness, grouping, and generated counts.
| File | Description |
|---|---|
workflow_workbench/rules.py |
Generates rule documentation from check metadata. |
tests/test_rules.py |
Tests generation and rule discovery. |
README.md |
Adds a generated rules summary. |
docs/well-formedness.md |
Documents checks and findings. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| def rules() -> tuple[Rule, ...]: | ||
| """Every exported check, in the order `checks.__all__` declares them.""" | ||
| out = [] | ||
| for name in checks.__all__: |
Comment on lines
+53
to
+62
| `check()` returns `CoherenceFinding`, a `str` subclass. The sentence is the finding; the fields | ||
| are there so a caller — or a coding agent using `check()` as its acceptance test — can branch | ||
| without parsing English. | ||
|
|
||
| ```python | ||
| f = spec.check(strategy)[0] | ||
| f.check # which rule above produced it | ||
| f.about # the node, 'source->target' for an edge, or '' for the whole design | ||
| f.blocking # False only for a `NOT CHECKED — …` stated gap | ||
| ``` |
… docstrings
The README said what the library is FOR and never what it checks. A reader could not
tell whether `coherence_check()` was three trivial rules or eleven real ones, and the
data language — the whole reason the design is inspectable — was never written down.
TWO GENERATED TABLES, in the README where they were asked for:
the data language every word you declare a design with, grouped
well-formedness every rule coherence_check() enforces, split by whether it
needs a strategy
python3 -m workflow_workbench.reference --write
python3 -m workflow_workbench.reference --check
NO DESCRIPTION IS AUTHORED. Every cell is the first line of the thing's own docstring,
or — for the two unions, which have none of their own — the names of their members.
A table cannot claim what the code does not say.
What IS authored is the GROUPING: which word sits under "boxes" and which under
"wires". That is curation, it is source, and it is a literal in `reference.py`. The
source/derived split is the point, so there is a test asserting GROUPS carries names
only.
THE README OPENING NOW NAMES THE FAILURE, NOT THE AUDIENCE. Boris's framing: an agent
left to run free produces work that RUNS AND IS INCOHERENT — each piece locally fine,
the whole not adding up. That is a reason to care; "for developing workflows with an
AI coding agent" was only a statement of who it is for. Four problems, one declaration
answering all four: incoherence, strategies that can only be compared and not asserted,
reuse, and visibility.
⚠️ The visualization claim is narrowed to what is true. pydantic-graph DOES emit
mermaid — `build_mermaid_graph` in `graph_builder.py` — so "they have none" would have
been false. Measured: it is not exported from their `__init__`, and it takes a BUILT
graph's internals, so every implementation must exist first. Two honest differences
then, not a feature list: ours reads the declaration, and ours can draw two strategies
at once.
Rebased onto the coherence-finding branch, because these tables name the method and
that branch renames it. #6 is stacked on #5.
ALSO FIXED AT SOURCE, not worked around:
· `_Start`'s first docstring line ended mid-sentence on "so `mypy` can narrow a", and
`_End`'s was "See `_Start`." Both are now standalone sentences, because the table
lifts that line verbatim. `test_every_first_docstring_line_stands_alone` makes a
wrapping summary a test failure rather than a fragment in the README.
· the union rows needed their pipes ESCAPED — a bare `|` is a column separator, so
`StepSpec | JoinSpec | DecisionSpec` rendered as three empty cells. Invisible in
the source string; the test asserts the rendered row.
· the generated prose said "`NodeSpec(...)` is a `TypeError`", which
`test_the_docs_use_the_current_api` greps for and flags. Reworded at the source
rather than exempting the README — that lint exists because the README's own
example once stopped running and nothing said so, and the README is the last file
that should be exempt from it. It has now caught me twice in this change, both
times correctly.
`docs/well-formedness.md` is deleted: the tables live in the README now, and two homes
for one generated table is the duplication this repo keeps a drift test to avoid.
Six ways this can rot were introduced one at a time, and each went red: hand-editing a
README row, a docstring changing without regeneration, a new export left out of the
table, a stale exclusion naming something no longer exported, GROUPS growing a
hand-written description, and the unions losing their pipe escaping.
245 tests pass. `parity --check` and `reference --check` both green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
borisdev
force-pushed
the
well-formedness-rules
branch
from
October 1, 2026 16:39
07bd73b to
d7d8abf
Compare
Comment on lines
+148
to
+150
| for name in checks.__all__: | ||
| if not name.startswith("check_"): | ||
| continue |
borisdev
added a commit
that referenced
this pull request
Oct 6, 2026
…— and five rounds of review (#5 #6 #7 #8) Contains #5, #6 and #7 — the stack is linear, so this one merge brings all four. - #5 `coherence_check()` returns `CoherenceFinding`, so an agent can branch instead of regex - #6 both reference tables in the README, generated from the package's own docstrings - #7 the glossary, linked from the prose, with a test that kills dead links - #8 the thesis, `examples/contestable.py`, `StepSpec.problem`, the implement-a-workflow skill Merged as a MERGE COMMIT, not a squash, so the eight commits stay in history. ## Five rounds of Copilot review, worked 20 inline comments were on record unread when consolidation started. Across five rounds: 14 live / 2 stale on round one, 5 on round two (three of them my own regression), pickle on round three, a stale docstring on round four, and two previously-missed on round five.⚠️ **Every single finding was a CHECK narrower than its claim, not code that was wrong.** The dead-link regex skipped malformed links; the "every rule" table enumerated 11 of 12; the retired -token map had no `.check(`; the compatibility oracle's five string operations omitted pickle; the rule-count guard scanned one phrasing in one file; a UI test drove one of three render surfaces. The code was mostly right. The things asserting it was right were the defects. 266 -> 278 tests. Both generators green. Bundle reproduces byte-for-byte from `npm run build`. ## One finding deliberately open Nested findings propagate an `about` relative to the child design, so it is not resolvable by a caller holding only the parent. Both readings are right about different things, and reconciling them means `about` carrying a path across a boundary — a semantic change to the field. Tracked in #9 with the limit documented in-code at both sites rather than left overstating.
Owner
Author
|
Contained in #8, merged as a merge commit — all eight commits are in |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Stacked on #5 — base is
coherence-finding, because these tables namecoherence_check()and that branch is where the rename lands. Merge #5 first.The README said what the library is for and never what it checks. A reader could not tell whether
coherence_check()was three trivial rules or eleven real ones — and the data language, the whole reason the design is inspectable at all, was never written down anywhere.Two generated tables, in the README
coherence_check()enforces, split by whether it needs a strategyNo description is authored. Every cell is the first line of the thing's own docstring — or, for the two unions, which have no docstring of their own, the names of their members. A table cannot claim what the code does not say.
What is authored is the grouping: which word sits under "boxes" and which under "wires". That's curation, it's source, and there's a test asserting
GROUPScarries names only — so the table can never become half source and half derived.The README opening now names the failure, not the audience
Your framing: an agent left to run free produces work that runs and is incoherent — each piece locally fine, the whole not adding up. That's a reason to care. "For developing workflows with an AI coding agent" only said who it was for.
Four problems, one declaration answering all four: incoherence, strategies that can only be compared rather than asserted, reuse, and visibility.
pydantic-graph does emit mermaid —
build_mermaid_graphingraph_builder.py— so "they have none" would have been false. Measured: it isn't exported from their__init__, and it takes a built graph's internals, so every implementation must exist first. Two honest differences, not a feature list: ours reads the declaration, and ours can draw two strategies at once.Fixed at source, not worked around
_Start's first docstring line ended mid-sentence on "somypycan narrow a";_End's was "See_Start." Both are standalone sentences now, because the table lifts that line verbatim. A test makes a wrapping summary a failure rather than a fragment in the README.|is a column separator, soStepSpec | JoinSpec | DecisionSpecrendered as three empty cells. Invisible in the source string; the test asserts the rendered row.NodeSpec(...)is aTypeError, whichtest_the_docs_use_the_current_apigreps for. Reworded at source rather than exempting the README — that lint exists because the README's own example once stopped running and nothing said so, and the README is the last file that should be exempt. It caught me twice in this change, both times correctly.docs/well-formedness.mdis deleted — the tables live in the README now, and two homes for one generated table is exactly the duplication this repo keeps drift tests to avoid.Mutation-tested
Six ways this can rot, introduced one at a time, each went red:
--checkdrift testGROUPSgrows a hand-written description245 tests pass.
parity --checkandreference --checkboth green.🤖 Generated with Claude Code