Skip to content

path query: parallel per-file execution (path-cli 0.17.0)#138

Open
benbaarber wants to merge 5 commits into
ben/sync-4-parent-dirfrom
ben/query-1-parallel-scan
Open

path query: parallel per-file execution (path-cli 0.17.0)#138
benbaarber wants to merge 5 commits into
ben/sync-4-parent-dirfrom
ben/query-1-parallel-scan

Conversation

@benbaarber

@benbaarber benbaarber commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #136. Part of the query-acceleration work.

What

path query now evaluates cache documents on a thread pool:

  • PerFileStream / Decompose plans treat every document independently, so the whole per-file pipeline — read, parse, wrap, filter, render/pack — runs on rayon workers (query/mod.rs::for_each_file, chunked). Only ordered consumption stays on the main thread. Per-file partials cross threads as compact JSON bytes (jaq Vals are Rc-based, not Send); compiled filters are cached once per pool thread.
  • Slurp plans parallelize the parse/wrap phase only — the merged jaq array must be built on one thread.
  • emscripten (playground) keeps the fully sequential engine; the worker-only helpers are cfg-gated off that target.

Measured (medians of 5, M-series)

Real cache — 97 docs / 114 MB / 46k steps:

Query before after speedup
length 898 ms 292 ms 3.1×
map(select(.dead_end)) | length 957 ms 309 ms 3.1×
map(select(.step.actor | startswith("agent:"))) | length 976 ms 310 ms 3.1×
.[] | .step.id (stream) 926 ms 303 ms 3.1×
--source claude 'length' 548 ms 200 ms 2.7×
map(.step.actor) | unique (slurp) 1021 ms 856 ms 1.2×
sort_by(.step.timestamp) | .[:5] | map(.step.id) (slurp) 1037 ms 878 ms 1.2×

Even synthetic cache — 60 docs / 56 MB / 132k steps (no skew):

Query before after speedup
length 966 ms 206 ms 4.7×
map(select(.dead_end)) | length 1042 ms 219 ms 4.8×
.[] | select(.dead_end) | .step.id 1063 ms 221 ms 4.8×
map(.step.actor) | unique (slurp) 1236 ms 1092 ms 1.1×

Re-measured 2026-07-23: the original real-cache numbers were taken on a
220 MB cache whose single 110 MB document (48% of the bytes) floored the
parallel wall at 2.0–2.1×; that document has since been deleted. The
synthetic table is unchanged.

Output unchanged

Every benchmark run diffed old-vs-new stdout: byte-identical in all cases (both caches). Unit tests additionally pin the parallel building blocks (render_file, partials_file + unpack_partials + finish_decompose) byte-for-byte against the sequential engine, including compact/pretty/raw modes and the zero-file Decompose rule; the sequential engine remains pinned against slurp by the existing tests.

Verification

  • scripts/quality_gates.sh: 7/7 pass
  • cargo check --target wasm32-unknown-emscripten -p path-cli: clean (no new warnings; worker-only helpers cfg-gated)
  • path-cli: 481 tests pass

View with Codesmith Autofix with Codesmith
Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.

PerFileStream and Decompose treat every document independently, so
their whole per-file pipeline (read, parse, wrap, filter, render/pack)
now runs on rayon workers, chunked to keep output order, warning order,
and error precedence byte-identical to the sequential scan. Partials
cross threads as compact JSON bytes since jaq values are Rc-based;
compiled filters are cached per pool thread. Slurp plans parallelize
the parse/wrap phase only, and the emscripten build stays sequential.

Real 97-doc/220MB cache: streamed/decomposed queries 1.3s -> 0.64s
(2.0-2.8x, floored by one 110MB doc); even synthetic cache ~4.7x.
Slurp queries ~1.1x. Outputs verified byte-identical across both
caches, plus unit tests pinning the parallel building blocks to the
sequential engine.
@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown

🔍 Preview deployed: https://4291e5f8.toolpath.pages.dev

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant