path query: parallel per-file execution (path-cli 0.17.0)#138
Open
benbaarber wants to merge 5 commits into
Open
path query: parallel per-file execution (path-cli 0.17.0)#138benbaarber wants to merge 5 commits into
benbaarber wants to merge 5 commits into
Conversation
PerFileStream and Decompose treat every document independently, so their whole per-file pipeline (read, parse, wrap, filter, render/pack) now runs on rayon workers, chunked to keep output order, warning order, and error precedence byte-identical to the sequential scan. Partials cross threads as compact JSON bytes since jaq values are Rc-based; compiled filters are cached per pool thread. Slurp plans parallelize the parse/wrap phase only, and the emscripten build stays sequential. Real 97-doc/220MB cache: streamed/decomposed queries 1.3s -> 0.64s (2.0-2.8x, floored by one 110MB doc); even synthetic cache ~4.7x. Slurp queries ~1.1x. Outputs verified byte-identical across both caches, plus unit tests pinning the parallel building blocks to the sequential engine.
|
🔍 Preview deployed: https://4291e5f8.toolpath.pages.dev |
# Conflicts: # CLAUDE.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #136. Part of the query-acceleration work.
What
path querynow evaluates cache documents on a thread pool:PerFileStream/Decomposeplans treat every document independently, so the whole per-file pipeline — read, parse, wrap, filter, render/pack — runs on rayon workers (query/mod.rs::for_each_file, chunked). Only ordered consumption stays on the main thread. Per-file partials cross threads as compact JSON bytes (jaqVals areRc-based, notSend); compiled filters are cached once per pool thread.Slurpplans parallelize the parse/wrap phase only — the merged jaq array must be built on one thread.Measured (medians of 5, M-series)
Real cache — 97 docs / 114 MB / 46k steps:
lengthmap(select(.dead_end)) | lengthmap(select(.step.actor | startswith("agent:"))) | length.[] | .step.id(stream)--source claude 'length'map(.step.actor) | unique(slurp)sort_by(.step.timestamp) | .[:5] | map(.step.id)(slurp)Even synthetic cache — 60 docs / 56 MB / 132k steps (no skew):
lengthmap(select(.dead_end)) | length.[] | select(.dead_end) | .step.idmap(.step.actor) | unique(slurp)Re-measured 2026-07-23: the original real-cache numbers were taken on a
220 MB cache whose single 110 MB document (48% of the bytes) floored the
parallel wall at 2.0–2.1×; that document has since been deleted. The
synthetic table is unchanged.
Output unchanged
Every benchmark run diffed old-vs-new stdout: byte-identical in all cases (both caches). Unit tests additionally pin the parallel building blocks (
render_file,partials_file+unpack_partials+finish_decompose) byte-for-byte against the sequential engine, including compact/pretty/raw modes and the zero-file Decompose rule; the sequential engine remains pinned against slurp by the existing tests.Verification
scripts/quality_gates.sh: 7/7 passcargo check --target wasm32-unknown-emscripten -p path-cli: clean (no new warnings; worker-only helpers cfg-gated)Need help on this PR? Tag
/codesmithwith what you need. Autofix is disabled.