feat(cli): self-syncing cache — `p cache sync`, query auto-sync, manifest artifact index by benbaarber · Pull Request #131 · empathic/toolpath

benbaarber · 2026-07-14T15:43:55Z

Makes the local cache fill itself instead of relying on manual p import. Supersedes #128 (closed when its base ben/query merged as #114); since then the branch was rebased onto the merged query work and gained the query auto-sync, manifest-index, parent-dir scoping, Copilot, and share-fast-path rounds, plus post-review hardening (privacy strip coverage, manifest locking, rotation-proof claude fingerprints, sync progress/interruptibility).

path p cache sync [types…] enumerates sessions across the installed harnesses (claude / gemini / codex / opencode / cursor / pi / copilot) and derives into the cache only what's new or changed. Change detection is stat-level — source mtime + size for file-backed providers, the DB row's updated-at for opencode/cursor — so a no-op sync reads no session bodies and finishes in milliseconds. Claude fingerprints cover the whole session chain (max segment mtime + summed sizes via claude_chain_stamp): Claude Code rotates to a new file on continuation while the chain keeps its oldest segment's id, so statting the head file alone would freeze the fingerprint at the first rotation and sync would never see later turns.

The manifest (~/.toolpath/sync.json, atomic 0600 writes) is an artifact index, not just a cache inventory: a record without a cache_id means "known, not materialized", so p cache rm downgrades records instead of orphaning them and out-of-band deletions self-heal on the next sync. Writers serialize on an advisory lock (sync.json.lock — a stable inode, since the manifest itself is rename-replaced) and every write is a locked read-merge-save of only the records that run produced, so concurrent invocations union instead of clobbering. Sync checkpoints every 10 writes and derives newest-first, so an interrupted first sync keeps nearly everything and spent its time on the sessions that matter most; pending work reports progress on stderr (live <type> done/total on a TTY, a line every 25 items otherwise — no-op syncs stay silent). Sync owns refresh semantics (overwrites what it re-derives; manual p import keeps error-on-hit), and the cache is an archive, not a mirror — sessions deleted upstream keep their documents.

path query syncs its scope before reading — --source claude syncs only claude, --id prefixes narrow likewise, a bare cache-wide query syncs every type, and --input-only queries never touch the cache. Quiet unless something was actually ingested, a sync failure degrades to querying the cache as-is, and --no-sync opts out. This is the piece that makes the cache an implementation detail: a new user can run path query with no setup and get their sessions.

--parent-dir <dir> / -d on p cache sync and path query restricts ingestion (and the query's reads) to artifacts under a directory. Stat gate first, always: unchanged+cached artifacts skip before any scope check; only artifacts that would cost a derive pay it, with one-line cwd peeks for codex/copilot memoized into the manifest. Claude project matching happens in slug space, and claude derives now source path.base from the session's recorded cwd instead of the lossy dir slug.

Everything that writes the cache records it — every p import flow (--session, picker multi-select, --all, most-recent fallbacks) and share upsert the manifest (claude imports key their record by the rotation-stable chain head), so sync never re-derives what they just produced. share uploads straight from the cache when the picked session is unchanged since its last sync, statting just that one artifact. Maximal ingest, strip at egress: thinking blocks are always cached for claude and gemini; the privacy decision moves to share / p export pathbase, which strip thinking from uploads by default with --include-thinking to opt in — the strip also scrubs sub-agent turns embedded in delegations, recursively. Local cache and resume fidelity keep everything.

toolpath-gemini 0.6.1: new PathResolver::list_session_entries, and peek_session_id is now bounded (4 KiB prefix scan with full-parse fallback) — this is what lets sync fingerprint gemini sessions without reading chat bodies.
toolpath-claude 0.12.1: ClaudeConvo::session_chain is now public (rides the cached chain index list_conversations already builds) — this is what lets sync fingerprint whole chains without reading bodies.

Breaking (pre-1.0): p import pi --all now emits one Path document per session (previously a single combined Graph); p import gemini loses --include-thinking (thinking is always ingested; stripping moved to egress). path-cli 0.15.0 → 0.16.0.

Tests: full scripts/quality_gates.sh green (fmt, shellcheck, clippy, test, doc, examples, site); new coverage for sync ingestion, query auto-sync, the manifest index, parent-dir scoping, rotation re-sync, delegation stripping, and progress/ordering.

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

`path p cache sync [types…]` enumerates sessions across the installed agent harnesses (claude/gemini/codex/opencode/cursor/pi, or just the named ones) and derives into the cache only what is new or changed. The manifest at ~/.toolpath/sync.json maps artifact type → session id → {project?, cache_id, last_activity?, message_count, synced_at}; a session re-derives only when its fingerprint (last_activity + message_count) differs. Manifest writes are atomic (temp + rename, 0600) and checkpointed per type; sync overwrites what it re-derives (refresh semantics) and never deletes — the cache is an archive, not a mirror. Enumeration reuses cmd_share::gather_sessions; per-session derivation goes through new *_with variants of the cmd_import session helpers so listing and derivation share provider managers (and tests can inject fixture resolvers). Real-data numbers: 34 sessions / 129 MB of Claude JSONL → 5.7 s cold, 0.9 s incremental no-op. path-cli 0.15.0 → 0.16.0 (+ toolpath-cli shim bump).

Sessions are just the first artifact kind sync and share operate on; the row type shouldn't bake that in. Fields keep their session-specific names until a second artifact kind actually lands.

ArtifactType (in sync.rs, pub, clap::ValueEnum) replaces the Harness/HarnessArg pair and is now the single enum naming artifact sources everywhere: p cache sync types, share/resume --harness, resume's source inference, and cmd_import's cache-id prefixes (the five wrap_paths_* clones collapse into one wrap_paths keyed on it). It lives ungated so emscripten builds of cmd_import still see it; the sync engine sits behind the cfg in a private submodule. ArtifactRow sheds its session-specific shape: harness → artifact_type, project → path, and message_count is gone — last_activity alone is the sync fingerprint, and the manifest record slims to {path?, cache_id, last_activity?, synced_at}. Pi's listing reports session start rather than last activity, so collect_pi now stats the session file's mtime; without that (or the old message_count), a growing pi session would never re-sync.

ArtifactType will grow non-session kinds (git, github), and those must be unrepresentable where the CLI means "an agent runtime": you can't resume into a git repo. Harness (cmd_share.rs, ValueEnum) names the six runtimes and is what share/resume --harness and resume's picker, availability checks, invocation, and projection take; it maps into the general enum via artifact_type(), with ArtifactType::harness() as the partial inverse. Sync, the manifest, import prefixes, and ArtifactRow stay on ArtifactType.

SyncRecord regains message_count as an Option, giving session fingerprints a second signal alongside last_activity. It stays a harness-only notion: sync_rows strips it via artifact_type.harness() so future non-session artifact kinds record None, and ArtifactRow carries it as Option<usize> populated by the six harness collectors. The share picker shows the msgs column again as a side effect.

Change detection previously compared last_activity + message_count, both of which come from listings that parse every session body — the fingerprint cost as much as the thing it was avoiding. Sync now enumerates ArtifactStubs whose fingerprint is the source file's mtime + size (claude: head segment via the bounded first-lines peek of list_conversations; codex: rollout stat, id from the stem's trailing uuid; pi: file stat, id from a one-line header peek) or the DB row's updated-at (opencode: header-only SELECT; cursor: composer headers, bubble-less drafts skipped, workspace-less composers now included). Real data: a 34-session no-op sync drops 0.9s → 11ms. Gemini still enumerates via its metadata listing (main files carry the session id inside the JSON); its fingerprint is a stat of the listed file, and a peek-level listing in toolpath-gemini is the follow-up. message_count stays in the manifest but is informational only, recorded at derive time from the source the derive already read (DerivedDoc.message_count), still restricted to harness types. The stamp is taken before the derive reads the source, so a write racing the derive re-syncs next run instead of going unnoticed.

…fest toolpath-gemini 0.7.0 adds PathResolver::list_session_entries — each session's listing id, inner sessionId, and backing file/dir path — and bounds peek_session_id to the first 4 KiB of a main chat file (identity fields lead the JSON; full parse only as fallback, guarded so a sessionId appearing after the messages key is never trusted from the prefix). list_sessions delegates unchanged. Sync's gemini stubs now use it, so no provider reads chat bodies to decide nothing changed. message_count leaves SyncRecord and DerivedDoc entirely: nothing ever read it from the manifest. ArtifactRow keeps its count for the share picker column, which is its only consumer.

list_session_entries is purely additive; for 0.y.z crates cargo treats z as the compatible slot and a y bump signals potential breakage, so ^0.6 consumers should get this for free.

Every session derive now carries a provenance ArtifactStub on DerivedDoc — identity plus a source stamp captured before the read, using the same per-provider lookups sync's enumeration uses (head-file stat for claude, entries listing for gemini with the canonical inner uuid, resolved rollout stat + uuid for codex, header SELECTs for opencode/cursor, peeked session file for pi). The cache-write sites in p import and share upsert the manifest via sync::record_stub, so the next sync sees those artifacts as unchanged instead of re-deriving them. The multi-select loops route through the derive_*_session_with helpers so picker imports are recorded too; bulk --all derives for claude/gemini/pi/codex carry no provenance and are picked up by the next sync instead. --no-cache paths record nothing. ArtifactType gains Git (7 variants): git imports are recorded — repo path plus the <repo-tag>-<graph-id> id — but never re-derived by sync, since repos aren't discoverable. Github and pathbase stay out of the manifest: remote services, not local artifact sources.

The bulk --all arms for claude/gemini/pi/codex and every most-recent fallback now enumerate ids and loop the derive_*_session_with helpers instead of calling the providers' bulk derive_project, so every cache write carries provenance and lands in the sync manifest. There was nothing to the bulk path performance-wise — read_all_* reads the same files one at a time — and wrap_paths plus the dead per-fn configs go with it. Breaking: p import pi --all now emits one Path document per session, consistent with every other provider (previously a single combined Graph via derive_graph).

A query now freshens exactly the slice of the cache it will read: --source claude syncs only claude, --id prefixes narrow likewise, a bare cache-wide query syncs every artifact type, and --input-only queries never touch the cache. Reporting is quiet unless something was actually ingested (synced claude: 3 new, 1 updated); a sync failure warns and degrades to querying the cache as-is; --no-sync opts out. Real data: a first-ever query over an empty cache ingests 32 sessions and answers in one command; the steady-state sync adds nothing measurable. Existing query integration tests pin $HOME to a shared empty sandbox so the implicit sync runs against nothing instead of the developer's real sessions.

SyncRecord.cache_id becomes optional: a record without one is known but not materialized. p cache rm downgrades records instead of orphaning them — fixing rm'd docs previously staying gone until their source changed — and sync verifies the doc file exists before skipping, so out-of-band deletions self-heal the same way. --parent-dir <dir> / -d on p cache sync and path query restricts ingestion (and the query's reads) to artifacts under a directory. The stat gate always runs first: unchanged+cached artifacts skip before any scope check; only artifacts already facing a derive get the constraint. Path-keyed providers prune whole projects before enumerating; opencode/cursor check the directory their cheap headers carry; codex — whose cwd lives inside the rollout — gets a one-line peek only when new/changed, memoized into the record's path so it happens at most once per artifact. Out-of-scope work tallies separately and never touches a materialized record's stamp, so staleness stays visible to the next in-scope sync. Claude's dir slugs are lossy ('/', '_', '.' all became '-'), so its scope matching happens in slug space, and its session derives stop passing the slug-derived project string into DeriveConfig — path.base now comes from the session's own recorded cwd.

Rebased ben/cache-sync onto the rewritten ben/query (now on main with the Copilot provider). Copilot becomes a full participant: an ArtifactType::Copilot + Harness::Copilot pair, stat-only enumeration over session-state/<id>/events.jsonl (dir name is the id), a bounded first-line peek for the session.start context cwd — memoized like codex's — per-session import loops with provenance, the share aggregator and resume arms restored onto the Harness-typed surfaces, and query auto-sync via --source copilot.

sync::fresh_cache_id answers "is this artifact's cached doc current?" — a fresh stat matches the manifest stamp and the doc file exists — using the same stub enumeration sync itself trusts. share_explicit checks it before deriving: on a hit it uploads the cached JSON as-is (a derive would reproduce the same bytes) and says so on stderr; a grown session or an evicted doc steps around the fast path and derives as before. --no-cache skips the check entirely.

The cache is the archive and now always holds the fullest derivation: claude and gemini ingest thinking unconditionally (breaking: p import gemini loses --include-thinking), while the privacy decision moves to the egress surfaces — share and p export pathbase strip thinking from uploads by default with --include-thinking to opt in there. Local projection keeps everything, so resume fidelity improves. This also dissolves the review's top finding: sync can no longer clobber a flag-derived doc because per-doc derive variance no longer exists. The other nine review findings: - p import --all warns-and-skips unreadable sessions again (all five per-session providers) instead of aborting the batch - importing an artifact the implicit query-sync already cached is a no-op with a friendly message, gated on real (non-None) stamps so git re-imports still honor error-on-hit - pi project scoping compares in its dir-encoded space, so hyphenated real paths match --parent-dir (mirror of claude's slug-space fix) - the copilot cwd peek scans a bounded number of lines and accepts top-level cwd keys, matching toolpath-copilot's own tolerance - a derive no longer clobbers the memoized peeked cwd in the record - claude sessions with no recorded cwd fall back to the caller's project for path.base instead of deriving baseless - gemini's peek runs the serde fallback even when the prefix scan declined on a small file (sub-agent dedup stays correct) - the toolpath-pi workspace pin returns to 0.6.1 (rebase damage) - sed-mangled doc comments and the stale six-harness count fixed

…rgeted share stat - strip_thinking also scrubs sub-agent turns embedded in delegations (gemini folds sub-agent chat files in, each turn with its own thinking), recursing into nested delegations - manifest writers serialize on an advisory lock (sync.json.lock); every write is a locked read-merge-save and sync checkpoints merge only the records the run wrote, so concurrent invocations union their records instead of clobbering each other - sync's unchanged gate no longer lets an all-None stamp vouch for a record (mirrors record_is_current) - share's fresh-cache fast path stats its one artifact directly (stamp_one) instead of enumerating the whole artifact type - ArtifactType::parse delegates to clap's ValueEnum instead of duplicating the name table

Claude Code rotates to a new JSONL file on continuation (plan-mode exit, resume, fork) while the chain keeps its oldest segment's id — appends land in the newest file, so statting the head file froze the fingerprint at the first rotation and sync never saw later turns. - toolpath-claude 0.12.1: ClaudeConvo::session_chain is now public (rides the cached chain index list_conversations already builds) - all three stamp sites (enumeration, import provenance, share fast path) share claude_chain_stamp: max segment mtime + summed segment sizes — exactly the files read_conversation merges, so fingerprint and derived doc move in lockstep across rotation-behavior changes - import normalizes its manifest key to the chain head so importing a successor-segment id records the artifact sync will look for

The doc quality gate (-D rustdoc::private-intra-doc-links) rejects public docs linking private items.

github-actions · 2026-07-14T17:04:15Z

🔍 Preview deployed: https://318cdfdb.toolpath.pages.dev

The first sync on a machine with years of sessions used to grind silently and, if interrupted, forget everything in the in-progress type. - progress on stderr whenever a sync has pending work: a live \r-updating '<type> done/total' line on a TTY, a plain line every 25 items otherwise; no-op syncs stay silent - manifest checkpoints every 10 writes instead of once per type, so an interrupted run keeps nearly everything it derived (out-of-scope peek memoizations included) - derives run newest-first, so partial progress covers the sessions the user most likely wants

akesling

This feels like it should really be at least three PRs. Harness/type refactoring, query logic update for cache, sync system, and maybe more?

Please break those out so it's easier to more effectively review. Maybe stack them relative to each other for simplicity?

akesling · 2026-07-16T14:10:11Z

+/// thinking included; documents leaving the machine drop it unless the
+/// caller opts in).
+#[cfg(not(target_os = "emscripten"))]
+pub(crate) fn strip_thinking(graph: &mut toolpath::v1::Graph) {


Why would we want to strip thinking, etc.?

I believe it's something we supported previously that carried over, I don't see any reason to keep it around though.

akesling · 2026-07-16T14:13:46Z

+//! are agent-session providers), compares each against the sync
+//! manifest at `$CONFIG_DIR/sync.json`, and derives + caches only what
+//! is new or changed. Change detection is stat-level: the fingerprint
+//! is the source file's mtime + size (or the database row's updated-at


Do we properly handle the case where mtime is "off"/unset?

in that case the doc would be rederived every time

akesling · 2026-07-16T14:14:19Z

+//! for the SQLite-backed providers), so deciding "nothing changed"
+//! never reads session bodies. Artifacts deleted upstream keep both
+//! their cache document and their manifest record — the cache is an
+//! archive, not a mirror.


What does it mean for something to be an archive, not a mirror?

Just claude being zesty, an unnecessary repetition of what was said before the em dash

akesling · 2026-07-16T14:16:05Z

+/// the provenance of each derived document so the write can be
+/// recorded in the manifest.
+#[derive(Debug, Clone)]
+pub(crate) struct ArtifactStub {


Would something like "ArtifactRef" or "ArtifactHandle" be a more recognizable/accurate name? I'm unsure what "stub" means here.

changed to ArtifactRef

akesling · 2026-07-16T14:18:14Z

+pub(crate) use engine::*;
+
+#[cfg(not(target_os = "emscripten"))]
+mod engine {


This feels like it should be broken out to its own file instead of being a nested module here.

now in sync/engine.rs

akesling · 2026-07-16T14:19:46Z

+    /// every '-' to '/', so a decoded project string is wrong for any
+    /// real path containing a hyphen. Compare in the encoded space,
+    /// where '/' boundaries are '-'.
+    fn pi_project_in_scope(project: &str, parent_dir: &Path) -> bool {


We're doing a lot of provider-specific work here. Who owns caching/sync for providers? It feels to me like we have an odd division here. If something changes in one of the provider crates, it may not get updated here. Perhaps there should be a reasonable central set of caching-related traits (or something derived from the existing system, perhaps?) which allows the central sync logic to not worry about the low-level details.

…erivation Review feedback on #131: there's no reason to strip thinking from Pathbase uploads. The cache still ingests maximally; --include-thinking on share / p export pathbase is gone along with the recursive delegation scrub.

Review feedback on #131: "stub" didn't say what the type is. It's a reference to an artifact — identity plus stat-level fingerprint.

Review feedback on #131: the nested module had grown to ~1800 lines inside a 2000-line file. Pure move; sync.rs keeps the artifact type/ref layer and re-exports the engine unchanged.

…urce trait Review feedback on #131: the engine was matching on ArtifactType in four places (enumerate, derive, stamp, scope), so a provider change meant editing central sync logic. sync/sources.rs now defines the seam — enumerate / stamp / derive / peek_dir / in_scope, one impl per provider, constructed from the HarnessBundle — and sync/engine.rs drives it without naming providers. Git maps to no source (recorded by p import, never discovered), replacing the runtime error path.

benbaarber · 2026-07-17T16:56:44Z

Superseded by the stacked series #133 → #134 → #135 → #136, split per review. The top of the stack is tree-identical to this branch's head (apart from a CHANGELOG restructure), and all review feedback here is addressed in it: thinking-stripping removed, ArtifactStub → ArtifactRef, and the engine split into sync/engine.rs + sync/sources.rs with provider-specific code behind a per-provider ArtifactSource trait.

benbaarber added 16 commits July 13, 2026 12:46

refactor(cli): rename SessionRow → ArtifactRow

6f7191e

Sessions are just the first artifact kind sync and share operate on; the row type shouldn't bake that in. Fields keep their session-specific names until a second artifact kind actually lands.

chore(gemini): 0.7.0 → 0.6.1 — additive change, patch slot for pre-1.0

37b6bc9

list_session_entries is purely additive; for 0.y.z crates cargo treats z as the compatible slot and a y bump signals potential breakage, so ^0.6 consumers should get this for free.

docs: pre-1.0 additive changes bump the patch slot

21b9b48

benbaarber marked this pull request as draft July 14, 2026 15:56

benbaarber added 3 commits July 14, 2026 12:24

fix(gemini): drop intra-doc link to private peek_session_id

8c890b0

The doc quality gate (-D rustdoc::private-intra-doc-links) rejects public docs linking private items.

benbaarber marked this pull request as ready for review July 14, 2026 19:33

benbaarber requested a review from akesling July 14, 2026 19:33

benbaarber assigned akesling Jul 14, 2026

akesling reviewed Jul 16, 2026

View reviewed changes

akesling assigned benbaarber and unassigned akesling Jul 16, 2026

benbaarber added 2 commits July 16, 2026 11:41

docs(cli): trim redundant archive-not-mirror clauses

f4df3c9

benbaarber added 4 commits July 16, 2026 11:42

refactor(cli): rename ArtifactStub to ArtifactRef

433e5b1

Review feedback on #131: "stub" didn't say what the type is. It's a reference to an artifact — identity plus stat-level fingerprint.

refactor(cli): move the sync engine out of sync.rs into sync/engine.rs

222507d

Review feedback on #131: the nested module had grown to ~1800 lines inside a 2000-line file. Pure move; sync.rs keeps the artifact type/ref layer and re-exports the engine unchanged.

docs(cli): drop stale strip-at-egress comments

c113ceb

benbaarber closed this Jul 17, 2026

Uh oh!

Conversation

benbaarber commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

akesling left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

benbaarber commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

benbaarber commented Jul 14, 2026 •

edited

Loading

github-actions Bot commented Jul 14, 2026 •

edited

Loading