diff --git a/README.md b/README.md index 2cb6c14..dd509df 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,38 @@ # workflow-workbench -Workflow Workbench is a high-level wrapper around Pydantic Graph Builder for developing workflows -with an AI coding agent. +**Let an AI coding agent build a workflow unsupervised and it produces code that works and is +incoherent.** Not broken — that you would notice. Incoherent: a fan-out whose results are +silently dropped, two wires crossed between values of the same type, a stage nobody implemented. +It runs, it returns something of the right shape, and nothing downstream can tell. -Specify the workflow and its data contracts, inspect its diagram, then have the agent implement -the steps. Bind alternative implementations as strategies and compare them through simple -evaluation battles. +Workflow Workbench is a declaration layer over Pydantic Graph Builder. You write the workflow's +shape and its data contracts as **data**, before any step exists — which is what makes that class +of defect findable: + +```python +spec.coherence_check() # 11 well-formedness rules, 7 of them with nothing implemented +spec.diagram() # a picture of the same declaration +spec.render(strategy) # refuses outright if anything blocks +``` + +So the agent gets an acceptance test it cannot talk its way past, and you get a drawing of the +design before you read a line of its code. + +Four problems, and the same declaration answers all four: + +| | | +|---|---| +| **An agent's output works and is incoherent.** Each piece is locally fine; the whole does not add up. | `coherence_check()` — 11 well-formedness rules, 7 needing nothing implemented | +| **A reasoning strategy cannot be asserted correct — only compared.** There is no right answer to diff against, so "better" is an empirical question. | `eval_battle()` — same cases, same evaluators, plus a replicate arm as the noise floor | +| **Complexity grows unless pieces are reused.** Two arms that differ in one stage should say so, not be two files. | the data language: declare a role once, bind it many ways; `SubgraphBinding` reuses a whole child design as one node | +| **You cannot see what you built.** | `diagram()` and `diff_diagram()`, from the declaration alone | + +On that last one, honestly: Pydantic Graph **can** emit mermaid — `build_mermaid_graph` in +`graph_builder.py`. Two differences, not a long list. It takes a BUILT graph's internals, so every +implementation must exist first; ours reads the declaration, so the picture arrives before the +code. And ours can draw **two strategies at once**, greying what they share and highlighting what +differs, which is a question about a comparison rather than about a graph. -The specification keeps the workflow understandable and gives the agent explicit constraints. Pydantic Graph executes the workflow; Pydantic Evals evaluates its results. Built on [Pydantic Graph](https://ai.pydantic.dev/graph/) and @@ -108,6 +133,46 @@ Stages 2 and 5 are what a specification buys, and neither needs a second strateg implementation per step still gets a drawing before it is written and a refusal when one is missed. +## What `coherence_check()` enforces + + +**11 rules.** `coherence_check()` returns one finding per violation and an empty list for a clean design; `render()` refuses on any finding that blocks. + +**7 need no implementations at all** — runnable the moment `nodes` and `edges` are written. + +| check | rule | +|---|---| +| `check_names` | Node names must be unique — `render()` uses them as graph node ids. | +| `check_reachable` | Every node reachable from START, and every node able to reach END. | +| `check_variables` | Per edge: the variable it carries must be an output of its source and an input of its target. | +| `check_step_arity` | A step body receives exactly ONE value, so a node cannot consume two inputs at once. | +| `check_decisions` | `when` appears exactly on the edges leaving a decision, and nowhere else. | +| `check_transform_edges` | A transform edge is fixed (`apply=`) or a variation point (bound) — exactly one. | +| `check_fan_out_rejoins` | Everything a fan-out produces must reach a join before it reaches END. | + +**4 more once a strategy exists**, checking the implementations against the roles they fill. + +| check | rule | +|---|---| +| `check_bindings` | The strategy binds exactly the declared VARIATION POINTS — no missing, no extra. | +| `check_implementations` | Each bound CALLABLE is callable and takes exactly one positional argument (`ctx`). | +| `check_subgraphs` | Every child design used as a node implementation fits the node it is bound to. | +| `check_variable_types` | Each implementation returns the type its role is declared to produce. | + + +Every one of these exists because it caught something that otherwise **ran and returned a +plausible answer**. Each check's docstring in [`checks.py`](workflow_workbench/checks.py) carries +the measured case that produced it. + +**These are structural checks, not a proof of correctness.** A step that returns its input +untouched satisfies every rule above and still does nothing — that is the boundary between what a +specification checks and what an evaluation measures, which is why `eval_battle` exists. + +A finding is a `CoherenceFinding`: a `str` subclass, so it reads as the sentence it is, carrying +`check`, `about` and `blocking` so an agent can branch on structure rather than parse English. A +`NOT CHECKED — …` finding is a **stated gap**, not a pass, and does not block `render()`. + + ## The same example, in five stages ### 1. Declare the nodes, the named values, and the edges @@ -134,6 +199,53 @@ checker can catch `compose` being wired to the wrong one when there is only one A name can. Edge fields are keyword-only and `carries` is required — four interchangeable-looking slots are one transposition away from a graph that is wrong and runs. +#### The data language + + +A design is **data** — tuples of these, in a class body. Nothing executes, which is what lets `coherence_check()` and `diagram()` read it before a single step is written. + +**Values** — What flows. Named, so a mis-wiring is visible when the types are identical. + +| | | +|---|---| +| `VariableSpec` | A named, typed value that may flow along an edge. | + +**Boxes** — Every box the design declares. Only a step takes an implementation. + +| | | +|---|---| +| `StepSpec` | A semantic role with a typed contract. Deliberately implementation-free. | +| `JoinSpec` | The one thing that can combine several arrivals into one value. | +| `DecisionSpec` | A router. Sends the value down one branch, chosen by its TYPE. | +| `NodeSpec` | `StepSpec` \| `JoinSpec` \| `DecisionSpec` | + +**Wires** — How values move. The kind of edge is the kind of movement. + +| | | +|---|---| +| `EdgeSpec` | One wire: `source -> target`, carrying `carries`. | +| `MapEdgeSpec` | Fan out: `carries` is a collection, and the target runs ONCE PER `delivers`. | +| `TransformEdgeSpec` | A cheap SYNCHRONOUS reshape that happens ON THE WIRE, creating no node. | + +**Endpoints** — The graph's own boundary, declared like anything else. + +| | | +|---|---| +| `START` | The graph's entry. | +| `END` | The graph's exit. | + +**The design, and what fills it** — One design, many competing sets of implementations. + +| | | +|---|---| +| `GraphSpec` | Subclass it, declare `nodes` and `edges`. That is the whole interface. | +| `StrategySpec` | A complete Bindable -> implementation mapping. One competitor. | +| `SubgraphBinding` | A whole child design — `GraphSpec` + `StrategySpec` — used as ONE node's implementation. | +| `Bindable` | `StepSpec` \| `TransformEdgeSpec` | + +The two unions are annotations, not classes you instantiate — calling either one raises `TypeError`. They exist so a signature can say *any declared box*, or *anything a strategy must bind*, and have it type-check. + + ### 2. Check it and draw it, before implementing anything ```python diff --git a/tests/test_reference.py b/tests/test_reference.py new file mode 100644 index 0000000..fedc841 --- /dev/null +++ b/tests/test_reference.py @@ -0,0 +1,136 @@ +"""The README's two tables are DERIVED, and this is what stops them drifting. + +Same shape as `test_parity.py`, for the same reason: `.claude/rules/spec-as-code.md` — an +unchecked convention drifts back within a month, so the check IS the rule. + +⚠️ Three halves, and only together do they mean anything: + · the generated blocks match the generator + · the generator sees EVERY export — a table perfectly in sync with a generator that reads nine + of eleven checks is perfectly in sync and wrong + · every cell is the thing's own words, so a table cannot say what the code does not +""" +from __future__ import annotations + +import inspect +import subprocess +import sys +from pathlib import Path + +import workflow_workbench as ww +import workflow_workbench.checks as checks +from workflow_workbench.reference import ( + GROUPS, + NOT_THE_LANGUAGE, + language_markdown, + rules, + rules_markdown, + vocabulary, +) + +ROOT = Path(__file__).resolve().parent.parent + + +def test_both_readme_blocks_are_regenerated_from_reference_py() -> None: + """Edit a docstring, regenerate, commit both. Editing the README alone turns this red.""" + proc = subprocess.run([sys.executable, "-m", "workflow_workbench.reference", "--check"], + cwd=ROOT, capture_output=True, text=True, timeout=120) + assert proc.returncode == 0, proc.stdout + proc.stderr + + +def test_every_export_is_either_in_the_language_or_explicitly_excluded() -> None: + """⛔ THE COMPLETENESS GUARD, and the one that matters most. + + A new type added to `__all__` and forgotten is the silent failure here: the table stays in + sync with the generator and quietly under-reports the language. Excluding something is fine — + saying nothing about it is not, which is why `NOT_THE_LANGUAGE` carries a reason per name. + """ + grouped = {w.name for _, _, words in vocabulary() for w in words} + assert grouped, "the vocabulary is empty — every assertion here would pass vacuously" + unaccounted = set(ww.__all__) - grouped - set(NOT_THE_LANGUAGE) + assert not unaccounted, ( + f"exported but neither in the table nor explained: {sorted(unaccounted)}. Add it to " + f"GROUPS, or to NOT_THE_LANGUAGE with the reason a reader does not need it.") + assert not (grouped & set(NOT_THE_LANGUAGE)), "a word cannot be both shown and excluded" + + +def test_the_exclusions_all_still_exist() -> None: + """A stale exclusion is worse than none: it names something nobody will notice is gone, and + the guard above goes quiet on whatever takes that name next. Copied from + `test_parity.py`'s exemption check, which exists for exactly this.""" + missing = [n for n in NOT_THE_LANGUAGE if n not in ww.__all__] + assert not missing, f"excluded from the table but no longer exported: {missing}" + + +def test_every_check_in_the_module_reaches_the_rules_table() -> None: + public = {n for n in dir(checks) + if n.startswith("check_") and callable(getattr(checks, n))} + assert public, "found no checks at all" + assert public == {r.check for r in rules()}, ( + "a check_* function is missing from checks.__all__, so the table cannot see it") + + +def test_no_cell_is_invented_prose() -> None: + """⛔ The rule that makes both tables trustworthy. If a cell could be hand-written, the table + could say something the code does not do — the whole failure a derived document removes.""" + for r in rules(): + first = inspect.getdoc(getattr(checks, r.check)).split("\n")[0].strip() + assert r.rule == first, f"{r.check}: table text is not its docstring's first line" + assert r.rule in rules_markdown(), f"{r.check}: its rule never reaches the table" + + for _, _, words in vocabulary(): + for w in words: + if w.is_union: + continue + first = inspect.getdoc(getattr(ww, w.name)).split("\n")[0].strip() + assert w.what == first, f"{w.name}: table text is not its docstring's first line" + + +def test_every_first_docstring_line_stands_alone() -> None: + """⚠️ The invariant the table rests on, and it is not free — `_Start`'s first line used to + end mid-sentence on 'so `mypy` can narrow a', which the table would have printed verbatim. + A docstring whose first line wraps is a source bug once anything lifts it.""" + for _, _, words in vocabulary(): + for w in words: + if w.is_union: + continue + assert w.what.endswith((".", "!")), ( + f"{w.name}: first docstring line does not end a sentence — it wraps, and the " + f"table would print the fragment. Put the summary on one line.") + + +def test_a_union_renders_its_members_and_not_pythons_docstring() -> None: + """`inspect.getdoc` on a `X | Y` alias returns 'Represent a PEP 604 union type', which says + nothing about this library. And the members must be PIPE-ESCAPED or markdown reads them as + column separators — a bug invisible in the source string and visible only in the table.""" + rendered = {w.name: w.what for _, _, words in vocabulary() for w in words if w.is_union} + assert set(rendered) == {"NodeSpec", "Bindable"}, "the two unions must both be shown" + assert "PEP 604" not in language_markdown() + for name, cell in rendered.items(): + assert r"\|" in cell, f"{name}: unescaped pipe would break the markdown table" + assert f"| `{name}` | {cell} |" in language_markdown() + + +def test_the_rules_table_counts_come_from_the_checks() -> None: + rs = rules() + md = rules_markdown() + assert f"**{len(rs)} rules.**" in md + assert f"**{sum(1 for r in rs if not r.needs_strategy)} need no implementations" in md + + +def test_needs_strategy_is_derived_and_matches_what_actually_runs() -> None: + """⚠️ `check_transform_edges` takes `strategy` and tolerates `None`, so it belongs with the + design-only group. Getting that wrong would tell a reader a design cannot be checked until it + is implemented — the opposite of the pitch.""" + design_only = {r.check for r in rules() if not r.needs_strategy} + assert "check_transform_edges" in design_only + assert {"check_bindings", "check_implementations", "check_variable_types", + "check_subgraphs"} == {r.check for r in rules() if r.needs_strategy} + + +def test_the_authored_part_is_only_the_grouping() -> None: + """GROUPS carries names and headings. If it ever carried a DESCRIPTION, the table would be + half source and half derived — which is the state this whole module exists to avoid.""" + for title, blurb, names in GROUPS: + assert isinstance(title, str) and isinstance(blurb, str) + assert all(isinstance(n, str) for n in names), ( + "GROUPS must hold names only — a description here would not be derived") diff --git a/workflow_workbench/reference.py b/workflow_workbench/reference.py new file mode 100644 index 0000000..299dc55 --- /dev/null +++ b/workflow_workbench/reference.py @@ -0,0 +1,226 @@ +"""The README's reference tables, DERIVED. SOURCE is the package's own docstrings. + +Two tables, one generator: + + the data language every word you declare a design with + well-formedness every rule `coherence_check()` enforces + + python3 -m workflow_workbench.reference # print both + python3 -m workflow_workbench.reference --write # rewrite the README's blocks + python3 -m workflow_workbench.reference --check # exit 1 if either is stale + +⛔ THIS MODULE INVENTS NO DESCRIPTIONS. Every cell is the first line of the thing's own +docstring, or — for the two unions — the names of their members. So a table cannot claim +something the code does not say. `.claude/rules/spec-as-code.md`: a document is either source or +derived, and mixing them is the whole failure mode. + +⚠️ What IS authored here is the GROUPING — which word belongs under "boxes" and which under +"wires". That is a curation decision and it is source, which is why it is a literal below and why +`test_reference.py` asserts every export is either grouped or deliberately excluded. A new type +that silently fails to appear would make the table under-report the language. +""" +from __future__ import annotations + +import inspect +import sys +import typing +from dataclasses import dataclass + +import workflow_workbench as ww +from workflow_workbench import checks + +__all__ = ["Word", "Rule", "vocabulary", "rules", "language_markdown", "rules_markdown"] + + +# ── the data language ─────────────────────────────────────────────────────────────────────── +# +# AUTHORED: the grouping and the order. Nothing else. + +GROUPS: tuple[tuple[str, str, tuple[str, ...]], ...] = ( + ("Values", "What flows. Named, so a mis-wiring is visible when the types are identical.", + ("VariableSpec",)), + ("Boxes", "Every box the design declares. Only a step takes an implementation.", + ("StepSpec", "JoinSpec", "DecisionSpec", "NodeSpec")), + ("Wires", "How values move. The kind of edge is the kind of movement.", + ("EdgeSpec", "MapEdgeSpec", "TransformEdgeSpec")), + ("Endpoints", "The graph's own boundary, declared like anything else.", + ("START", "END")), + ("The design, and what fills it", "One design, many competing sets of implementations.", + ("GraphSpec", "StrategySpec", "SubgraphBinding", "Bindable")), +) + +#: Exported, and deliberately NOT in the table above, with the reason. A reader of the data +#: language does not need these to write a design — but an unexplained omission is how a table +#: comes to under-report, so each one is named. +NOT_THE_LANGUAGE: dict[str, str] = { + "SpecError": "raised BY the language, not part of writing one", + "CoherenceFinding": "what a check returns — the second table's subject", + "blocking": "a filter over findings", + "NOT_CHECKED": "the prefix marking a stated gap", + "diagram": "output, not declaration", + "diff_diagram": "output, not declaration", + **{n: "one check — the second table" for n in checks.__all__ if n.startswith("check_")}, +} + + +@dataclass(frozen=True) +class Word: + """One word of the data language: what you write, and what it is.""" + + name: str + what: str + is_union: bool + + +def _describe(name: str) -> tuple[str, bool]: + """A word's own description, or its members if it is a union. + + ⚠️ A `X | Y` alias has no docstring of its own — `inspect.getdoc` returns `UnionType`'s, + which is "Represent a PEP 604 union type" and says nothing about this library. Rendering the + MEMBERS is both truer and incapable of drifting: it is the definition. + """ + obj = getattr(ww, name) + members = typing.get_args(obj) + if members: + # ⚠️ `\|`, escaped. A bare pipe is a COLUMN SEPARATOR in a markdown table, so the + # union rendered as three empty columns — visible only in the rendered table, + # which is why the test below asserts on the rendered row and not on this string. + return " \\| ".join(f"`{m.__name__}`" for m in members), True + doc = inspect.getdoc(obj) + if not doc: + raise SystemExit(f"{name} has no docstring — the table's only source") + return doc.split("\n")[0].strip(), False + + +def vocabulary() -> tuple[tuple[str, str, tuple[Word, ...]], ...]: + out = [] + for title, blurb, names in GROUPS: + words = tuple(Word(n, *_describe(n)) for n in names) + out.append((title, blurb, words)) + return tuple(out) + + +def language_markdown() -> str: + out = [ + "A design is **data** — tuples of these, in a class body. Nothing executes, which is " + "what lets `coherence_check()` and `diagram()` read it before a single step is written.", + "", + ] + for title, blurb, words in vocabulary(): + out += [f"**{title}** — {blurb}", "", "| | |", "|---|---|"] + out += [f"| `{w.name}` | {w.what} |" for w in words] + out += [""] + # ⚠️ Do NOT write the retired constructor form here, even to say it is retired: + # `test_the_docs_use_the_current_api` greps every prose doc for it and the README is not + # exempt, deliberately. Describing the behaviour reads better anyway. + out += ["The two unions are annotations, not classes you instantiate — calling either one " + "raises `TypeError`. They exist so a signature can say *any declared box*, or " + "*anything a strategy must bind*, and have it type-check."] + return "\n".join(out) + + +# ── the well-formedness rules ─────────────────────────────────────────────────────────────── + +@dataclass(frozen=True) +class Rule: + """One well-formedness rule: the check that enforces it, and what it says about itself. + + `needs_strategy` is derived from the SIGNATURE, not a hand-kept list — a check requiring a + `strategy` cannot run on a design with nothing implemented, and that is the most useful thing + to know before reading the rest of the row. + """ + + check: str + rule: str + needs_strategy: bool + + +def _needs_strategy(fn) -> bool: + """A REQUIRED `strategy`. `| None` means it tolerates having none, so it still runs on a bare + design — `check_transform_edges` is exactly that case and must not be grouped with the four + that genuinely cannot.""" + p = inspect.signature(fn).parameters.get("strategy") + return p is not None and "None" not in str(p.annotation) + + +def rules() -> tuple[Rule, ...]: + out = [] + for name in checks.__all__: + if not name.startswith("check_"): + continue + fn = getattr(checks, name) + doc = inspect.getdoc(fn) + if not doc: + raise SystemExit(f"{name} has no docstring — the table's only source") + out.append(Rule(name, doc.split("\n")[0].strip(), _needs_strategy(fn))) + return tuple(out) + + +def rules_markdown() -> str: + rs = rules() + design = [r for r in rs if not r.needs_strategy] + strategy = [r for r in rs if r.needs_strategy] + out = [ + f"**{len(rs)} rules.** `coherence_check()` returns one finding per violation and an empty " + f"list for a clean design; `render()` refuses on any finding that blocks.", + "", + f"**{len(design)} need no implementations at all** — runnable the moment `nodes` and " + f"`edges` are written.", + "", + "| check | rule |", + "|---|---|", + ] + out += [f"| `{r.check}` | {r.rule} |" for r in design] + out += ["", + f"**{len(strategy)} more once a strategy exists**, checking the implementations " + f"against the roles they fill.", + "", + "| check | rule |", + "|---|---|"] + out += [f"| `{r.check}` | {r.rule} |" for r in strategy] + return "\n".join(out) + + +# ── generation ────────────────────────────────────────────────────────────────────────────── + +BLOCKS = (("language", language_markdown), ("rules", rules_markdown)) + + +def _readme(): + import pathlib + return pathlib.Path(__file__).resolve().parent.parent / "README.md" + + +def main() -> int: + if "--check" not in sys.argv and "--write" not in sys.argv: + for _, fn in BLOCKS: + print(fn(), "\n") + return 0 + target = _readme() + text = target.read_text() + rc = 0 + for tag, fn in BLOCKS: + start, end = f"", f"" + if start not in text or end not in text: + print(f"README.md: {tag} markers missing") + rc = 1 + continue + head, rest = text.split(start, 1) + stale, tail = rest.split(end, 1) + body = fn() + if "--write" in sys.argv: + text = f"{head}{start}\n{body.strip()}\n{end}{tail}" + elif stale.strip() != body.strip(): + print(f"README.md: the {tag} table is stale. Regenerate:\n" + f" python3 -m workflow_workbench.reference --write") + rc = 1 + else: + print(f"README.md: {tag} table matches reference.py") + if "--write" in sys.argv: + target.write_text(text) + print("README.md: both tables rewritten from reference.py") + return rc + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/workflow_workbench/spec.py b/workflow_workbench/spec.py index 3cbc5b5..b77de5f 100644 --- a/workflow_workbench/spec.py +++ b/workflow_workbench/spec.py @@ -46,8 +46,11 @@ class SpecError(Exception): class _Start: - """The graph's entry. A class, not a bare `object()`, so `mypy` can narrow a - `NodeSpec | _Start | _End` union — a bare sentinel makes every `edge.source` lookup unprovable.""" + """The graph's entry. + + ⚠️ First line stands alone, because `reference.py` lifts it into the README's vocabulary + table. A class, not a bare `object()`, so `mypy` can narrow a `NodeSpec | _Start | _End` + union — a bare sentinel makes every `edge.source` lookup unprovable.""" __slots__ = () @@ -56,7 +59,9 @@ def __repr__(self) -> str: class _End: - """The graph's exit. See `_Start`.""" + """The graph's exit. + + A class for the same reason `_Start` is.""" __slots__ = ()