feat(crush): code crusher — Go AST (Tier A) + tree-sitter multi-language (Tier B)

[initializ-mk]

What

Adds a dedicated code crusher for detect.SourceCode (previously routed to the extractive text crusher). It keeps a file's shape — package clause, imports, type/const/var decls, and every function signature — and elides function bodies, offloading each interior in place. Two tiers:

• Tier A — Go, in the core via the stdlib go/parser + go/ast. No third-party dependency; AST-accurate brace matching, so a {/} inside a string or comment never fools it.
• Tier B — Python / TypeScript / JavaScript / Java / C / C++ via tree-sitter, in a separate codelang module.

Scope note (router)

The router.go change in this PR is only SourceCode → code. GitDiff → diff and SearchResults → search were already wired on main by #4 and #6 — they are not touched here. (git diff origin/main...HEAD -- router/router.go shows the single SourceCode case flip.)

Tier B isolation (module, not build tags)

tree-sitter needs CGO + the C grammars, so codelang is its own Go module (github.com/initializ/ctxzip/codelang, own go.mod). The core module's go.mod stays bbolt-only — the CGO/tree-sitter dependency is fully isolated and opt-in. It plugs into the core crusher via a small crush.CodeCrusher.Fallback seam (nil → extractive text, so default behavior is unchanged and nothing is forced onto CGO):

code := crush.NewCodeCrusher()        // Go via go/ast (no CGO)
code.Fallback = codelang.NewCrusher() // Python/TS/JS/Java/C/C++ via tree-sitter

How Tier B works

Detect language from content signals → try candidate grammars in likelihood order (a dialect is tried before its base only when its exclusive syntax fires — TS's interface/type-annotations, C++'s std::/template, not shared tokens like class) → parse-and-verify (reject if ERROR/MISSING nodes cover >15% of bytes; try the next) → elide function/method bodies, keeping braces (Python's brace-less block offloaded whole after the :). A wrong-but-clean parse still elides real body nodes, so compression stays valid regardless of the label.

Reversibility

Position-preserving and byte-exact for both tiers. Each body's interior is offloaded and replaced in place by a <<ctxzip:HASH>> marker occupying exactly those bytes, so expanding every marker reconstructs the source byte-for-byte — asserted for Go and for all six Tier-B languages.

Deliberate deviation (called out for review)

The generic error-vocabulary floor is intentionally not applied to code. It exists so a log/tool-output crusher never drops the error line a user is about to ask about; source is different — bodies are saturated with error/Errorf, so the floor would pin nearly every function and defeat compression, and bodies are reversible anyway. MustKeep vocabulary and query terms still protect a body.

Safety / tests

Fail-open (nil store / unparseable / sub-MinLines / per-body Put failure / unsupported language → text); deterministic; never empty.

• Core: 10 crusher tests (incl. the Fallback seam) + router assertion; make check green, deps unchanged.
• codelang (CGO): per-language elision with correct detection, byte-exact round-trip for all six, an adversarial TS-not-claimed-by-JS case, MustKeep protection, unsupported-language fallback, determinism, and the integration seam. go test ./... green.

End-to-end on a real Go file: 4320 → 3083 tokens (29%), 17 bodies elided and reversible, structure intact.

Follow-ups (per the plan)

More tree-sitter languages in codelang (Rust/Ruby/…) · ML prose + embedding scorer (HybridScorer.Embed) · Redis ccr.Store backend. C# was left out per scope.

[initializ-mk]

Pushed Tier B — tree-sitter multi-language support, bringing the code crusher to parity with headroom's code_compressor.

New: codelang module (Python / TypeScript / JavaScript / Java / C / C++)

A separate Go module (github.com/initializ/ctxzip/codelang, own go.mod) — deliberately, because tree-sitter needs CGO + the C grammars. Core stays bbolt-only (verified: core go.mod unchanged; the CGO dependency is fully isolated). Only hosts that want multi-language code compression take it on.

codelang.Crusher implements crush.Compressor:

• Detect language from content signals; candidates tried in likelihood order (specific dialects first — TS over JS, C++ over C).
• Parse + verify: reject a parse whose ERROR/MISSING nodes cover >15% of bytes and try the next candidate. A wrong-but-clean parse still elides real body nodes, so compression stays valid and reversible regardless of the label.
• Elide function/method bodies (> MinBodyLines), keeping imports, type decls, class headers, and every signature. Braces kept; interior offloaded in place (Python's brace-less block offloaded whole after the :).

Core change (small): the Fallback seam

crush.CodeCrusher gains a Fallback Compressor field (nil → text). Go still uses the stdlib go/ast path; the injected tree-sitter crusher handles the rest:

code := crush.NewCodeCrusher()
code.Fallback = codelang.NewCrusher()

Default behavior is unchanged (nil fallback → text), so this PR doesn't force CGO on anyone.

Same invariants, tested

• Byte-exact reversibility — expand every marker → source, asserted per language (all six).
• Correct detection per language (each subtest asserts the right grammar), MustKeep protection, unsupported-language (Ruby) → text fallback, determinism, and the integration seam (Python → tree-sitter, Go → go/ast).
• Error-vocab floor still intentionally not applied to code; MustKeep + query terms protect a body.

make check green (core, unchanged deps); cd codelang && go test ./... green (CGO). C# was left out per scope.

Note: this grew PR #7 beyond Tier-A — happy to split the codelang module into its own follow-up PR if you'd prefer to review Tier A independently.

[initializ-mk]

Review — code crusher (Go AST Tier A + tree-sitter Tier B)

Reviewed empirically from a local checkout: built both modules, ran the suites, and wrote adversarial round-trip probes for both tiers (then removed them). The reversibility invariant holds — proven byte-exact on adversarial input for both tiers. The design is clean: Tier B's CGO/tree-sitter lives in a separate codelang module so the core stays pure-Go (bbolt-only). Findings are about scope/docs and one detection nuance — no correctness or reversibility defects.

🟠 MEDIUM #1 — Undocumented scope: the router change activates three crushers, not one

router.go doesn't just route SourceCode → code; it also flips GitDiff → diff and SearchResults → search (both previously went to text). Those crushers were merged earlier (#4/#6) but never wired — this PR wires all three. It's tested (router_test asserts all three) and the diff/search crushers are unchanged here (wiring only), so it's safe. But the title/body describe only the code crusher, so a changelog reader would miss that diff and search compression just went live. Call it out in the description.

🟡 LOW #2 — Stale "build-tagged / follow-up" wording (body + code comment)

The body says "Build-tagged tree-sitter for the rest (Tier B) is a follow-up," but Tier B ships in this PR (commit f50f296, the codelang module) and is isolated by a separate module, not build tags. crush/code.go repeats the "build-tagged … is a follow-up" phrasing. Update both — the actual (better) isolation story is module-level, and Tier B is not a follow-up.

🟡 LOW #3 — TS content can be claimed by the JavaScript grammar (detection quality)

Empirically, a TypeScript sample (with interface + : string/: number annotations) parsed as code_treesitter:javascript, not typescript — JS signal count outscored TS's higher weight, and the JS parse stayed under maxErrorRatio. Reversibility is unaffected (byte-exact confirmed), but TS-specific bodies (interface method signatures, typed arrow properties) can be missed → less compression. Consider trying TS before JS whenever any TS-only signal fires, or weighting TS-only signals above raw count.

Informational (not a finding): a valid Go file with no elidable bodies returns ok=false from compressGo and then gets text-crushed (extractive line-dropping on source). Reversible and matches the pre-PR SourceCode → text routing, so no regression — just slightly odd to extractively crush parseable Go. Fine to leave.

✅ Verified correct (empirically)

• Tier A (Go AST) byte-exact on braces inside interpreted strings, raw strings, and line/block comments; generics; nested closures; multibyte UTF-8; empty/small/protected bodies — the interior never leaked (AST brace-matching holds). Identical bodies dedup to one store entry yet all positions reconstruct. The error-floor deviation works (error-saturated bodies still elided). Structure (package/imports/types/signatures) preserved.
• Tier B (tree-sitter) byte-exact on TypeScript (braces in strings/template-literals/comments) and Python (indent blocks, braces in strings). collectBodies skips descent into an elided body → spans never overlap; the {/} brace guard validates before slicing; maxErrorRatio rejects wrong-grammar parses; fail-open to text.
• Module isolation: core go.mod has no tree-sitter/CGO dependency; codelang depends on core one-way (replace => ../). Core stays pure-Go; Tier B is opt-in via CodeCrusher.Fallback. Both modules build; all suites pass.

Verdict: changes requested (all non-blocking) — the reversibility core is solid; the asks are scope/doc clarity (#1/#2) and a detection-quality nit (#3).

[initializ-mk]

🟠 MEDIUM #1 — this change activates three crushers, but the PR describes only code. Alongside SourceCode → code, this switch also flips GitDiff → diff and SearchResults → search (both previously routed to text). Those crushers were merged in #4/#6 but never wired; this PR wires all three. It's tested and the diff/search crushers are unchanged here, so it's safe — but it's a real behavioral change for diff/search content that the title/body don't mention. Please call it out in the PR description/changelog.

[initializ-mk]

🟡 LOW #2 — stale wording. This says a build-tagged tree-sitter path "is a follow-up," but Tier B ships in this same PR as the codelang module (commit f50f296), and it's isolated by a separate module, not build tags (cleaner — the core go.mod has no CGO/tree-sitter dep at all). Update this comment (and the PR body) to describe the module-level isolation and that Tier B is included here, not deferred.

[initializ-mk]

Thanks — thorough review (the adversarial byte-exact probes especially). Addressed below.

🟠 MEDIUM #1 — I think this is a stale-checkout artifact, not a change in this PR

This PR's router.go diff only flips SourceCode → code; GitDiff → diff and SearchResults → search were already wired on main by #4 and #6. git diff origin/main...HEAD -- router/router.go:

@@ case detect.SourceCode:
-		// TODO: dedicated AST crusher. Route to extractive text for now …
-		return r.text
+		return r.code

and git show origin/main:router/router.go already has case detect.GitDiff: return r.diff and case detect.SearchResults: return r.search. So diff/search compression went live in #4/#6, not here — I suspect the local checkout was branched from a pre-#4/#6 main. Happy to add a note if you still want it, but I didn't want to document a behavioral change this PR doesn't actually make. (If your tree shows otherwise, let me know and I'll dig in.)

🟡 LOW #2 — fixed

Updated the crush/code.go CodeCrusher doc: the non-Go path is the tree-sitter codelang module (Tier B, shipped in this PR), isolated as a separate module (core go.mod stays CGO-free) — not a build-tagged follow-up. PR title/body already reflect Tier A + B.

🟡 LOW #3 — fixed

Reworked detection so a dialect is tried before its base only when its exclusive syntax is present. Each language now has a strong regex (exclusive markers) distinct from signals (candidacy); rank = strongHits*1000 + totalHits:

• TS ranks ahead of JS via interface/type-annotation strong hits even when JS-shared signals outnumber TS ones — your exact finding.
• Pure JS never becomes a TS candidate (TS candidacy no longer fires on shared function/const).
• Java/C aren't hijacked by C++ (dropped the shared class/#include from C++ candidacy; Java/Python strong markers outrank a bare class).

New TestCrusher_TSNotClaimedByJS locks it: a TS file with more JS signals than TS still detects as typescript and round-trips byte-exact. (A first weight-first-only attempt regressed Java — C++'s broad class signal preempted it — which is what drove the strong/shared split.)

All six languages detect correctly; codelang suite + core make check green.

[github-actions]

🔎 Grype — no fixable vulnerabilities ✅

[initializ-mk]

Re-review — commit c6b7f183 (detection rework)

Verified locally (both modules build; crush, router, codelang suites green; re-ran my own detection + round-trip probes). The code-level findings are resolved.

Resolved

• 🟡 LOW #3 — TS claimed by JS: fixed and verified. The strong/shared split (score = strongHits*1000 + totalHits, with TS candidacy now requiring TS-exclusive markers) works. I confirmed empirically: typescript, java, cpp, c, javascript each detect correctly, and a TS file with more JS signals than TS now detects as typescript and round-trips byte-exact via the typescript grammar. I also checked the C++ cross-signal edge you fixed — a Java file with :: method references (Object::toString, System.out::println) correctly detects as java, not cpp, because Java's strong markers outrank C++'s lone shared signal. Nice catch on the first-attempt Java regression.
• 🟡 LOW #2 (code half) — fixed. The crush/code.go doc now describes the module-level isolation (core keeps no CGO/tree-sitter dependency; Tier B is the injectable codelang module), not a build-tagged follow-up.

Still open (PR-description edits only — non-blocking)

• 🟠 LOW #1 — the PR body/title still describe only the code crusher. The router change also activates GitDiff → diff and SearchResults → search (previously text); please note that behavioral change in the description.
• 🟡 LOW #2 (body half) — the "## How" section still says "Build-tagged tree-sitter for the rest (Tier B) is a follow-up." Tier B ships in this PR as the module-isolated codelang; update that line to match the code comment you just fixed.

Both are quick PR-body edits (no commit needed). The reversibility core and detection logic are solid and empirically verified — once the description is squared away this is good to merge.

[initializ-mk]

Final re-review — all items resolved (and a correction on my part)

⚠️ Withdrawing MEDIUM #1 — my error, you were right

I claimed this PR also activated GitDiff → diff and SearchResults → search. That was a false positive: I diffed against a stale local main that predated the #4/#6 router wiring. Verified against a fresh base — git diff origin/main...HEAD -- router/router.go shows only the SourceCode → code flip (plus the code field/constructor); origin/main already routes diff and search. Your scope note in the body is correct. Apologies for the noise — that finding is withdrawn.

Resolved

• 🟡 LOW #2 — fixed (code + body). crush/code.go describes the module-level isolation, and the body's stale "build-tagged … follow-up" line is gone, replaced by an accurate "Tier B isolation (module, not build tags)" section with the Fallback seam.
• 🟡 LOW #3 — fixed and verified. Detection reworked with a strong/exclusive-marker split; I confirmed empirically that ts/java/cpp/c/js all detect correctly, a JS-heavy TS file now detects as typescript and round-trips byte-exact, and the :: method-reference edge stays java (not cpp).

Verdict: ✅ LGTM — good to merge

The reversibility core is proven byte-exact on adversarial input for both tiers (Go AST and tree-sitter), Tier B's CGO/tree-sitter is fully isolated in the codelang module (core go.mod stays bbolt-only), detection is robust, and every fail-open path is covered. Clean piece of work — nice module boundary and the empirical byte-exactness guarantee is exactly the right invariant to hold the line on.

(No CI on this repo; verified locally — both modules build, all suites green.)