Skip to content

path query: SQLite step index (path-cli 0.18.0)#139

Draft
benbaarber wants to merge 6 commits into
ben/query-1-parallel-scanfrom
ben/query-2-sqlite-index
Draft

path query: SQLite step index (path-cli 0.18.0)#139
benbaarber wants to merge 6 commits into
ben/query-1-parallel-scanfrom
ben/query-2-sqlite-index

Conversation

@benbaarber

@benbaarber benbaarber commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #138. Implements Approach A from docs/design/query-acceleration.md (committed here, with measured results).

What

A disposable SQLite accelerator at $CONFIG_DIR/index.db (query/index.rs):

  • Every wrapped step as a row — compact JSON, exactly what wrap_graph emits — plus VIRTUAL generated columns (dead_end, actor, source) with per-doc composite indexes.
  • Stat-gated freshness: each doc records its file's (mtime_ns, size); rows serve only on a fresh match, otherwise the file reparses and reindexes in place (stamp taken before the read, so racing writes re-derive next query). Cache files stay canonical: delete the index and it rebuilds lazily; a schema-version bump rebuilds automatically; TOOLPATH_QUERY_NO_INDEX=1 bypasses.
  • Write-through: cache::write_cached indexes what it writes (sync/import keep the index warm); p cache rm purges; full scans prune orphans.
  • Predicate recognition (plan.rs::RowPredicate, whole-predicate-or-nothing): .dead_end (and | not), .step.actor == "…", .step.actor | startswith("…"), .path.meta.source == "…", and and-conjunctions — from a leading map(select(P)) / .[] | select(P) / [.[] | select(P)] stage.
  • Two accelerated shapes: a filter that is exactly length or map(select(P)) | length collapses to SELECT count(*) (no step parsed at all); any recognized leading select prefilters rows in SQL before parsing, with the unchanged original filter still running on survivors.
  • Content-scoped queries (--kind/--project/--parent-dir) bypass the index (those filters live in the wrapping pass). --input/stdin are never indexed. emscripten build compiles without any of this.

Measured (real 97-doc / 114 MB / 46k-step cache, medians of 5)

Query baseline rayon #138 + index output
length 898 ms 292 ms 30 ms IDENTICAL
map(select(.dead_end)) | length 957 ms 309 ms 30 ms IDENTICAL
map(select(.step.actor | startswith("agent:"))) | length 976 ms 310 ms 169 ms IDENTICAL
.[] | select(.dead_end) | .step.id 937 ms 304 ms 311 ms IDENTICAL
map(.step.actor) | unique (slurp) 1021 ms 856 ms 881 ms IDENTICAL
group_by(.path.id) | map(length) | max (slurp) 1032 ms 879 ms 902 ms IDENTICAL
--source claude 'length' 548 ms 200 ms 28 ms IDENTICAL

Absorbed counts are flat (~30 ms) regardless of cache size/shape (same
on the 56 MB synthetic cache). Honest costs and limits: one-time index
build ~1.7 s on first query; index size ~250 MB beside the 114 MB
cache; a predicated stream lands at parity with #138's parallel scan
(row-fetch + per-row parse ≈ parallel whole-file parse), and
unpredicated slurps and streams stay within a few percent of #138.
rm ~/.toolpath/index.db reverts at any time.

Re-measured 2026-07-23: the cache previously held a single 110 MB
document (48% of the bytes); with it, the predicated dead-end stream
was slower than the parallel scan (724 ms vs 615 ms — its surviving
rows carried the fat diffs) and absorbed counts showed ~46× against
the fatter baseline. That document has since been deleted.

Output unchanged

The integration suite (tests/query.rs) asserts index-served output is byte-identical to TOOLPATH_QUERY_NO_INDEX=1 across every plan shape — absorbed counts, predicated decompose/stream, predicated and unpredicated slurp, --source scoping, --raw — on first (indexing) and second (serving) runs, plus staleness self-healing (file changed/removed/rewritten behind the index) and p cache rm purging. Unit tests cover the store (round-trip, stamps, replace, purge, retain, empty docs, version rebuild) and the recognizer (recognized shapes, conservative refusals, the in-code matches mirror).

Verification

  • scripts/quality_gates.sh: 7/7 pass
  • cargo check --target wasm32-unknown-emscripten -p path-cli: clean, no new warnings
  • path-cli: 371 unit + 41 query-integration tests pass

Also included: docs/design/query-acceleration.md (the design this implements, updated with measured results) and docs/design/graph-query-frontend.md (the facto/GraphJQ exploration it cross-references).


View with Codesmith Autofix with Codesmith
Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.

A disposable accelerator at $CONFIG_DIR/index.db: every wrapped step
as a row (compact, exactly what the jaq engine consumes) with generated
columns over the hot fields, stat-gated freshness per document, and
write-through from cache::write_cached so sync/import keep it warm.
Files stay canonical; deleting the index (or bumping its schema
version) just rebuilds it lazily. TOOLPATH_QUERY_NO_INDEX=1 bypasses.

Two recognized shapes accelerate: a filter that is exactly length /
map(select(P))|length with a whitelisted P collapses to SELECT
count(*) (1.23s -> 27ms on the real 220MB cache, no step parsed), and
a recognized leading select prefilters rows in SQL before parsing
(predicated decompose 5x). Recognition is whole-predicate-or-nothing
and the unchanged original filter still runs on what survives, so
answers cannot change; integration tests assert index-served output is
byte-identical to the no-index path across every plan, including
staleness self-healing and p cache rm purging. Content-scoped queries
bypass the index. Slurp and byte-heavy stream queries ride 0.17.0's
parallel scan unchanged.

Also commits the query-acceleration design doc (with measured results)
and the graph-query-frontend exploration it links to.
@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown

🔍 Preview deployed: https://49cad047.toolpath.pages.dev

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant