path query: SQLite step index (path-cli 0.18.0)

CHANGELOG.md

@@ -2,6 +2,54 @@
 
 All notable changes to the Toolpath workspace are documented here.
 
+## SQLite step index for `path query` — 2026-07-21
+
+`path query` now keeps a disposable SQLite accelerator at
+`$CONFIG_DIR/index.db`: every wrapped step as a row (exactly what the
+jaq engine consumes), with generated columns over the hot fields and a
+stat-level freshness gate per document. The cache files stay canonical
+— delete the index and the next query rebuilds it lazily; a schema
+version bump rebuilds it automatically. `TOOLPATH_QUERY_NO_INDEX=1`
+bypasses it.
+
+What it accelerates (real 97-doc / 114 MB / 46k-step cache, medians;
+"rayon" = 0.17.0's parallel scan):
+
+- **Absorbed counts** — a filter that is exactly `length` or
+  `map(select(P)) | length` with a recognized `P` becomes
+  `SELECT count(*)`: 0.90 s → **30 ms** (~30×; no step is parsed at
+  all). Recognized `P` atoms: `.dead_end` (and `| not`),
+  `.step.actor == "…"`, `.step.actor | startswith("…")`,
+  `.path.meta.source == "…"`, and `and`-conjunctions of those.
+- **Predicated scans** — a leading `map(select(P))` / `.[] | select(P)`
+  prefilters rows in SQL before parsing: `map(select(.step.actor |
+  startswith("agent:"))) | length` 0.98 s → 169 ms (~6×; 310 ms under
+  rayon alone).
+- Unrecognized/slurp queries are unchanged (they ride 0.17.0's
+  parallel scan); a predicated stream lands at parity with it
+  (row-fetch + per-row parse ≈ parallel whole-file parse).
+
+Correctness: recognition is exact (whole-predicate or nothing) and the
+prefilter feeds the *unchanged* original filter, so pruned rows are
+ones the filter's own first stage would drop — the integration suite
+asserts index-served output is byte-identical to the no-index path
+across every plan, staleness self-healing included. Content-scoped
+queries (`--kind`/`--project`/`--parent-dir`) bypass the index.
+
+- **`path-cli`** (0.18.0):
+  - `query/index.rs`: the store — WAL, per-thread connections,
+    `(mtime_ns, size)` stamps, write-through from `cache::write_cached`
+    (sync/import keep the index warm), purge on `p cache rm`, orphan
+    pruning on full scans, version-gated self-rebuild. One-time build
+    cost on first query: ~1.7 s for the 114 MB cache; index size ~250 MB
+    (rows + hot-field indexes).
+  - `query/plan.rs`: `RowPredicate` — conservative recognizer for the
+    leading-`select` predicate and the absorbable-count shape, with an
+    in-code `matches` mirror used when a stale doc is reparsed.
+  - `TOOLPATH_QUERY_EXPLAIN=1` now also prints the index strategy
+    (`count absorbed into SQL (dead_end=true)`, `serving fresh docs,
+    prefilter …`, or `off`).
+
 ## Parallel `path query` execution — 2026-07-21
 
 `path query` now evaluates cache documents on a thread pool. For the

Cargo.toml

@@ -37,7 +37,7 @@ toolpath-github = { version = "0.6.0", path = "crates/toolpath-github" }
 toolpath-dot = { version = "0.5.0", path = "crates/toolpath-dot" }
 toolpath-md = { version = "0.7.0", path = "crates/toolpath-md" }
 toolpath-pi = { version = "0.6.1", path = "crates/toolpath-pi" }
-path-cli = { version = "0.17.0", path = "crates/path-cli" }
+path-cli = { version = "0.18.0", path = "crates/path-cli" }
 pathbase-client = { version = "0.2.0", path = "crates/pathbase-client" }
 
 reqwest = { version = "0.13", default-features = false, features = ["blocking", "json", "rustls"] }

crates/path-cli/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "path-cli"
-version = "0.17.0"
+version = "0.18.0"
 edition.workspace = true
 license.workspace = true
 repository = "https://github.com/empathic/toolpath"

crates/path-cli/src/cache.rs

@@ -83,6 +83,13 @@ pub(crate) fn write_cached(id: &str, doc: &Graph, force: bool) -> Result<PathBuf
         std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o600))
             .with_context(|| format!("chmod 0600 {}", path.display()))?;
     }
+    // Write-through to the query step index, so sync/import keep it warm
+    // and the next query serves this doc without reparsing it. Best-effort:
+    // the index is a disposable accelerator and self-heals via its stat gate.
+    #[cfg(not(target_os = "emscripten"))]
+    if let Err(e) = crate::query::index::index_written_doc(id, doc, &path) {
+        eprintln!("warning: query index not updated for {id}: {e:#}");
+    }
     Ok(path)
 }
 

crates/path-cli/src/cmd_cache.rs

@@ -62,6 +62,10 @@ fn run_rm(id: &str) -> Result<()> {
     if let Err(e) = crate::sync::evict_cache_id(id) {
         eprintln!("warning: sync manifest not updated: {e}");
     }
+    #[cfg(not(target_os = "emscripten"))]
+    if let Err(e) = crate::query::index::purge_id(id) {
+        eprintln!("warning: query index not updated: {e}");
+    }
     eprintln!("Removed {id}");
     Ok(())
 }