feat: graphify curate — durable human corrections that survive a rebuild

[cgmoore120]

Problem

Every edge in graph.json is extractor-derived, so a hand correction is transient by construction — in both directions, and silently.

A deleted edge comes back. The semantic cache is keyed by file content, not by graph state (cache.py:531). An unchanged doc keeps its cached edges forever, so the next graphify extract re-injects the edge you deleted and writes it with force=True (cli.py:2574) — past the shrink guard.

An added edge is destroyed. build_merge replaces per source_file (build.py:925-950): every source_file present in the new chunks is dropped from the base graph — nodes and edges — before merging. A human-authored edge whose source_file names a re-extracted file is dropped, and no extractor re-emits it. Nothing warns, and the node count grows while it happens, so the loss reads as a successful update.

I hit the second one on a 3,384-node Angular graph. I had verified four extracted edges against the source, deleted the three that were false, and added three that extraction had missed. Then I ran graphify update . to time it:

before:  3,384 nodes,  9,118 edges
after:   3,425 nodes,  9,148 edges     <- grew
lost:    3 verified edges, silently

The backup_if_protected dated folder from #834 saved me — but that is recovery, not persistence: it hands you a snapshot and lets the overwrite proceed. (The asymmetry is nasty: my deletions survived, because those edges came from .html files that update doesn't re-extract, while my additions died, because they were attributed to .ts files that it did. A partial wipe looks like success.)

save-result --outcome dead_end looks like the answer, but it is advisory — it informs a future reader through LESSONS.md and never gates extraction. So the graph and the lessons actively drift apart: the graph keeps asserting an edge the operator has already disproved, and keeps re-advertising it in Surprising Connections.

There is currently no user-owned override mechanism anywhere in the package (grep for curat|override|pin|deny: the word "curated" appears only in export.py, meaning "the labels file has a non-default community name").

Solution

A small, declarative graphify-out/curation.json, applied at the end of build_from_json — the single funnel through which build, build_merge, watch and the skill's agent path all construct the graph.

Applying it at build time rather than write time is the load-bearing choice: clustering, god nodes, surprising_connections and GRAPH_REPORT.md all see the corrected graph. A disproved edge stops being re-advertised, rather than merely being absent from graph.json.

# an edge the extractor keeps inventing
graphify curate deny page_not_found patient_search --relation semantically_similar_to \
  --reason "lexical collision on 'not found'; no shared service, model, or import"

# an edge verified in the source that extraction missed
graphify curate pin usage_table medicare_table --relation shares_data_with --score 0.95 \
  --source-file src/usage_table.py --source-location L45 \
  --reason "both read CareManagementReport via subReportLoaded"

graphify curate list     # review the overlay
graphify curate apply    # apply to graph.json now, without a rebuild

Semantics:

• Denies match the unordered pair (undirected storage canonicalizes endpoint order, so a deny written (a,b) must match an edge stored (b,a)), and honour _src/_tgt so a direction-restored edge still matches. Omitting --relation denies every edge between the pair.
• Pins are skipped, never invented, when an endpoint is missing — the overlay corrects the graph, it does not fabricate nodes (an invented endpoint would dangle and be silently swallowed by the dangling-edge drop).
• Denies run before pins, so one entry can deny a mistyped edge and pin the corrected one over the same pair.
• Re-application is idempotent.
• The two --no-cluster paths (cli.py, watch.py) never build a NetworkX graph, so they apply the overlay to the payload directly — otherwise a denied edge survives on exactly those paths.
• curation.json joins _BACKUP_ARTIFACTS (Feature: protect semantic and curated graphs from silent overwrite via dated-folder backup #834).
• Inert when absent — no behavior change for existing users. GRAPHIFY_NO_CURATION=1 disables it (mirrors GRAPHIFY_NO_BACKUP).

This complements save-result --outcome dead_end rather than replacing it: that records that a path was false, for a human reading LESSONS.md. Curation makes the graph itself stop asserting it.

Testing

23 new tests in tests/test_curation.py, including a regression for each failure mode:

• test_pinned_edge_survives_build_merge_replace_per_source — pins an edge whose source_file is the file that then gets re-extracted, and asserts it survives build_merge.
• test_denied_edge_stays_denied_when_extraction_reasserts_it — runs three builds where the extraction keeps asserting the edge (as the content-keyed cache does), and asserts it never lands.

Plus order-insensitivity, relation scoping, idempotency, deny-before-pin, missing-endpoint skip, malformed/future-schema/env-var handling, and --no-cluster payload coverage.

End-to-end on a real corpus: pinned an edge on src/alpha.py, modified src/alpha.py (forcing re-extraction of exactly that file), ran graphify update . — the pin survived and the deny held. A subsequent full graphify extract . (the cache-resurrection path) also held.

$ uv run --frozen pytest tests/ -q
3157 passed, 31 skipped

5 pre-existing failures (test_ollama_retry_cap.py, test_labeling.py::test_label_communities_batches_when_over_batch_size) fail identically on v8 without this change — verified by stashing.

CI gates: ruff clean; skillgen --check, --audit-coverage, --schema-singleton, --monolith-roundtrip, --always-on-roundtrip all OK; bandit reports nothing in the new code.

Notes for review

• I deliberately did not touch merge-driver (cli.py:1420), which unions two graphs and bypasses to_json, backup and the shrink guard. Applying the overlay there is defensible but it is a separate concern — happy to add it if you'd like it in scope.
• Related open issues this sits beside: Feature request: let the git-hook/watch rebuild honor a persisted clustering config (exclude-hubs) #1800 (hook rebuild doesn't honour persisted clustering config — the same "rebuild must honour operator intent" family; a shared overrides file might be the right long-term shape) and [Bug] build_merge severs edges from UNCHANGED files into re-extracted files (−12% doc↔code in one update cycle): endpoint nodes are replaced, the surviving edges dangle, and the dangling-edge drop silently eats them #1711 (build_merge severs edges from unchanged files whose endpoints were replaced — a different mechanism from this one, and not fixed here).
• Schema is versioned (version: 1) and a future/unknown version is ignored with a warning rather than half-applied.