feat(query): add relevance-aware scoped retrieval

[javargasm]

Summary

• Add relevance-aware graph retrieval that separates semantic query text from explicit memory, community, and god-node scopes.
• Make query, path, and explain deterministic and ambiguity-safe across CLI, MCP, and HTTP transports.
• Teach all 16 generated host skills the same retrieval contract from one canonical source, with strict generator and monolith drift guards.

Problem

Graphify could build a structurally valid graph and still return a noisy or misleading search result.

Saved Q: work-memory nodes competed with source code during ordinary queries. Broad BFS/DFS traversals discovered nodes without considering the requested token budget and only truncated the rendered text afterward. Common labels such as main(), Config, or handle() could resolve to whichever matching node happened to be encountered first, and CLI and MCP used different resolution paths. Community labels existed in .graphify_labels.json, but query retrieval could not use those names as scopes.

That combination made large graphs harder to navigate precisely:

• prior saved answers could dominate fresh code evidence;
• exact documentation or test matches could be displaced by lower-value code candidates;
• repeated runs over equivalent graphs could return different bounded subsets;
• overloaded symbols could silently select the wrong file;
• architecture hubs discovered during clustering were not addressable from normal queries;
• missing, external, and file-only source provenance could be rendered as if they were equivalent.

The issue is not graph construction. It is the retrieval policy between a valid graph and the answer presented to the user.

Root causes

Area	Previous behavior	Consequence
Memory	Saved query nodes were scored as ordinary graph nodes	Historical Q&A polluted default starting sets
Candidate ranking	Scope and relevance were not staged explicitly	Broad code matches could crowd out exact docs/tests
Traversal budget	BFS/DFS traversed first and truncated text later	Work and returned context were only loosely bounded
Symbol lookup	CLI and MCP used different first/fuzzy-match paths	Ambiguous labels could resolve inconsistently
Communities	Human labels lived in a sidecar outside query loading	Users could not scope by architecture community name
Provenance	Rendering did not distinguish every source state	Citations could imply ownership or precision that was unavailable
Skills	Host variants documented older fallback behavior independently	Agent behavior could diverge from the actual CLI/MCP contract

What changed

Relevance-aware retrieval

graphify query now separates semantic text from explicit control directives:

include:memory
community:<id|label>
god:<label|id>

Default retrieval is staged across code, documentation, tests, and likely communities. Saved work-memory remains available as a fallback when it is the only lexical evidence, but it does not compete with normal source results unless include:memory is intentionally present. A global relevance quota preserves strong cross-scope and cross-community matches so staging does not become a hard information silo.

Community filters load canonical names from the adjacent .graphify_labels.json artifact and continue to accept numeric IDs. God filters scope retrieval and traversal to a selected hub. Unknown filters return copy/pasteable suggestions instead of silently falling back to an unscoped search.

Deterministic, budget-aware traversal

BFS and DFS now apply a deterministic node bound derived from --budget before rendering. Neighbor ordering is stable, high-degree transit hubs remain guarded, and equivalent graphs no longer return different bounded node sets merely because edges were inserted in a different order.

The renderer can still truncate edge-heavy output at the final character budget, but traversal itself is now bounded rather than discovering an arbitrarily large subgraph first.

One ambiguity-safe resolver

CLI and MCP now share a resolver for exact node IDs, readable labels, source paths, and path-qualified identities:

<source-file>::<label>

Plain labels resolve only when the result is unambiguous. Otherwise:

• CLI path and explain print ranked candidates and exit with status 2;
• MCP/HTTP return the same diagnostic as successful tool-result text;
• callers rerun with the printed source-qualified identity or exact node ID.

The resolver uses graph-scoped exact/path/basename indexes. Source suffixes are checked only after narrowing by basename, avoiding the quadratic retained-memory amplification that would come from materializing every suffix of a graph-controlled path.

Truthful source display and bounded untrusted input

Source rendering now preserves the distinction between:

• known line spans;
• (file only) ownership;
• external/reference provenance from origin_file;
• (no source) concepts.

Graph-controlled output is sanitized for control, formatting, surrogate, and Unicode line-separator categories before it reaches CLI/MCP text. Query and sidecar processing are bounded:

Input	Bound
Query terms	First 32 normalized terms
Query term length	128 characters
Graph-local IDF cache	512 LRU entries
Source-qualified path	4,096 characters, 256 components, 255 characters/component
Community-label sidecar	1 MiB, 4,096 entries, 1,024 characters/label

Malformed or oversized label sidecars are rejected atomically and retrieval safely falls back to embedded names and numeric IDs.

Skills follow the implementation

All 16 host variants now receive one canonical query contract covering:

• explicit memory inclusion;
• community and god scopes;
• deterministic budgets;
• ambiguity handling across CLI and MCP;
• source-qualified identities;
• truthful source display;
• safe argument passing for user-controlled questions;
• limitations of the inline NetworkX fallback.

The generator renders 14 split variants plus the Aider and Devin monoliths. It verifies exact slot cardinality, byte-compares the canonical query section inside monoliths, retains the frozen-v8 historical guard for all surrounding content, and rejects missing, duplicated, or tampered insertion points.

Why this approach

The retrieval stages deliberately rerank the existing graph instead of changing clustering or inventing semantic edges. This keeps the trust boundary clear: extraction decides what exists, while retrieval decides what evidence should be considered first.

Memory is fallback-only rather than deleted because saved outcomes remain useful when the corpus has no other lexical match. A small global quota remains outside the preferred scope because architecture questions often cross community boundaries. Ambiguity is surfaced rather than automatically broken because a deterministic wrong answer is still wrong.

Community names are read from the canonical sidecar instead of copied into graph nodes during query time. That preserves the existing build artifact model and lets relabeling take effect without rebuilding graph topology.

Review map

The PR is organized as two independently reviewable commits:

Commit	Review first	Purpose
814a956	graphify/serve.py, CLI/security changes, focused tests	Runtime retrieval, resolution, bounds, provenance, and transport contracts
0065aa0	Canonical skill fragments, tools/skillgen/gen.py, tests/test_skillgen.py	Agent guidance, all host variants, generated artifacts, and drift guards

Review the canonical generator inputs before generated outputs. The generated graphify/skill*.md, split query references, and tools/skillgen/expected/** files are deterministic products checked by skillgen --check.

Size impact

This is intentionally one PR because runtime behavior and the instructions consumed by agents must land together. Splitting them would leave a window where installed skills invoke semantics the engine does not support, or where the engine ships without safe guidance for reserved directives and ambiguous identities.

Category	Additions	Deletions	Review lines
Authored implementation, canonical docs, and tests	2,671	967	3,638
Generated skills and snapshots	2,716	8,604	11,320
Total	5,387	9,571	14,958

Most deletion churn comes from replacing duplicated query runbooks in 16 host variants and their expected snapshots with one shorter canonical contract. This follows the repository precedent in #1737: one subsystem per commit, explicit generated-size accounting, and strong byte/test evidence inside a single large PR.

Validation

uv run pytest tests/ -q
3295 passed, 3 skipped

uv run python -m tools.skillgen --check
check OK: 134 artifact(s) match committed output and expected/.

uv run python -m tools.skillgen --audit-coverage
audit-coverage OK

uv run python -m tools.skillgen --schema-singleton
schema-singleton OK

uv run python -m tools.skillgen --monolith-roundtrip
monolith-roundtrip OK

uv run pre-commit run --all-files
skillgen --check: passed
Ruff: passed

uv lock --check
passed

git diff --check v8...HEAD
passed

Compatibility and intentional behavior changes

• Existing plain queries remain valid.
• Existing numeric community IDs remain valid, but durable instructions should prefer names because IDs can change after reclustering.
• Ambiguous CLI path/explain calls now fail with exit 2 instead of silently selecting a candidate.
• MCP/HTTP preserve text-result transport semantics and return ambiguity diagnostics without a transport error.
• Advanced directives are reserved control syntax. Agents append them only intentionally and do not forward literal directive discussions as executable query text.
• Missing source spans are not synthesized.

Out of scope

• No clustering or community-detection redesign.
• No new semantic edges or inferred source ownership.
• No change to saved-query persistence or reflection formats.
• No global installation change.
• External-URL rendering was not exercised on the real validation graph because it contains no external URL nodes; dedicated synthetic regressions cover external origin_file provenance.

Related work

• Builds on the constrained, graph-vocabulary-grounded query direction introduced in Constrained query expansion #998.
• Extends the lookup consistency work represented by fix: resolve_seed matches bare names against ()-decorated callable labels #1265 and [codex] fix node label lookup normalization #1353.
• Uses the generated-artifact review pattern from fix(skill): pin extraction source_file + root the full build so --update stops duplicating #1366 and the large-PR safety/evidence pattern from refactor: decompose extract.py and __main__.py into focused modules #1737.