path query: SQLite step index (path-cli 0.18.0)#139
Draft
benbaarber wants to merge 6 commits into
Draft
Conversation
A disposable accelerator at $CONFIG_DIR/index.db: every wrapped step as a row (compact, exactly what the jaq engine consumes) with generated columns over the hot fields, stat-gated freshness per document, and write-through from cache::write_cached so sync/import keep it warm. Files stay canonical; deleting the index (or bumping its schema version) just rebuilds it lazily. TOOLPATH_QUERY_NO_INDEX=1 bypasses. Two recognized shapes accelerate: a filter that is exactly length / map(select(P))|length with a whitelisted P collapses to SELECT count(*) (1.23s -> 27ms on the real 220MB cache, no step parsed), and a recognized leading select prefilters rows in SQL before parsing (predicated decompose 5x). Recognition is whole-predicate-or-nothing and the unchanged original filter still runs on what survives, so answers cannot change; integration tests assert index-served output is byte-identical to the no-index path across every plan, including staleness self-healing and p cache rm purging. Content-scoped queries bypass the index. Slurp and byte-heavy stream queries ride 0.17.0's parallel scan unchanged. Also commits the query-acceleration design doc (with measured results) and the graph-query-frontend exploration it links to.
|
🔍 Preview deployed: https://49cad047.toolpath.pages.dev |
# Conflicts: # CLAUDE.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #138. Implements Approach A from
docs/design/query-acceleration.md(committed here, with measured results).What
A disposable SQLite accelerator at
$CONFIG_DIR/index.db(query/index.rs):wrap_graphemits — plusVIRTUALgenerated columns (dead_end,actor,source) with per-doc composite indexes.(mtime_ns, size); rows serve only on a fresh match, otherwise the file reparses and reindexes in place (stamp taken before the read, so racing writes re-derive next query). Cache files stay canonical: delete the index and it rebuilds lazily; a schema-version bump rebuilds automatically;TOOLPATH_QUERY_NO_INDEX=1bypasses.cache::write_cachedindexes what it writes (sync/import keep the index warm);p cache rmpurges; full scans prune orphans.plan.rs::RowPredicate, whole-predicate-or-nothing):.dead_end(and| not),.step.actor == "…",.step.actor | startswith("…"),.path.meta.source == "…", andand-conjunctions — from a leadingmap(select(P))/.[] | select(P)/[.[] | select(P)]stage.lengthormap(select(P)) | lengthcollapses toSELECT count(*)(no step parsed at all); any recognized leadingselectprefilters rows in SQL before parsing, with the unchanged original filter still running on survivors.--kind/--project/--parent-dir) bypass the index (those filters live in the wrapping pass).--input/stdin are never indexed. emscripten build compiles without any of this.Measured (real 97-doc / 114 MB / 46k-step cache, medians of 5)
lengthmap(select(.dead_end)) | lengthmap(select(.step.actor | startswith("agent:"))) | length.[] | select(.dead_end) | .step.idmap(.step.actor) | unique(slurp)group_by(.path.id) | map(length) | max(slurp)--source claude 'length'Absorbed counts are flat (~30 ms) regardless of cache size/shape (same
on the 56 MB synthetic cache). Honest costs and limits: one-time index
build ~1.7 s on first query; index size ~250 MB beside the 114 MB
cache; a predicated stream lands at parity with #138's parallel scan
(row-fetch + per-row parse ≈ parallel whole-file parse), and
unpredicated slurps and streams stay within a few percent of #138.
rm ~/.toolpath/index.dbreverts at any time.Re-measured 2026-07-23: the cache previously held a single 110 MB
document (48% of the bytes); with it, the predicated dead-end stream
was slower than the parallel scan (724 ms vs 615 ms — its surviving
rows carried the fat diffs) and absorbed counts showed ~46× against
the fatter baseline. That document has since been deleted.
Output unchanged
The integration suite (
tests/query.rs) asserts index-served output is byte-identical toTOOLPATH_QUERY_NO_INDEX=1across every plan shape — absorbed counts, predicated decompose/stream, predicated and unpredicated slurp,--sourcescoping,--raw— on first (indexing) and second (serving) runs, plus staleness self-healing (file changed/removed/rewritten behind the index) andp cache rmpurging. Unit tests cover the store (round-trip, stamps, replace, purge, retain, empty docs, version rebuild) and the recognizer (recognized shapes, conservative refusals, the in-codematchesmirror).Verification
scripts/quality_gates.sh: 7/7 passcargo check --target wasm32-unknown-emscripten -p path-cli: clean, no new warningsAlso included:
docs/design/query-acceleration.md(the design this implements, updated with measured results) anddocs/design/graph-query-frontend.md(the facto/GraphJQ exploration it cross-references).Need help on this PR? Tag
/codesmithwith what you need. Autofix is disabled.