feat(cli): p cache sync — incremental ingestion with a manifest artifact index#134
feat(cli): p cache sync — incremental ingestion with a manifest artifact index#134benbaarber wants to merge 11 commits into
Conversation
…act index Adds path p cache sync [types…]: stat-level change detection against a manifest at ~/.toolpath/sync.json (atomic 0600 writes, advisory-locked read-merge-save, checkpointed every 10 writes, newest-first derives, stderr progress). Provider specifics live behind the ArtifactSource trait (sync/sources.rs: enumerate / stamp / derive, one impl per provider); the engine (sync/engine.rs) never matches on artifact type. Claude fingerprints cover the whole rotation chain via claude_chain_stamp. p import and share record what they write (provenance ArtifactRef on every session derive); p cache rm downgrades records to known-but-uncached; share uploads straight from the cache when the source is unchanged.
|
🔍 Preview deployed: https://09cff82e.toolpath.pages.dev |
| } | ||
|
|
||
| fn is_not_found_claude(err: &toolpath_claude::ConvoError) -> bool { | ||
| pub(crate) fn is_not_found_claude(err: &toolpath_claude::ConvoError) -> bool { |
There was a problem hiding this comment.
I feel like anything in the cmd_* modules probably shouldn't be used from outside. Also, should this move to a common place that's parameterized somehow given we have seven of these at this point? Maybe there should be a generic into or autoref to ConvoError on each type?
Perhaps instead of a bool it would be some general toolpath_convo::Error::SessionNotFound?
There was a problem hiding this comment.
First point is fixed now with changes to the PR below this on the stack. Likely these functions should move to toolpath_convo in some consolidated error struct in a follow up PR
# Conflicts: # CLAUDE.md # crates/path-cli/src/artifact.rs # crates/path-cli/src/cmd_import.rs # crates/path-cli/src/cmd_share.rs
# Conflicts: # crates/path-cli/src/cmd_import.rs
# Conflicts: # crates/path-cli/src/cmd_import.rs # crates/path-cli/src/cmd_share.rs
# Conflicts: # crates/path-cli/src/cmd_import.rs
| use crate::config::config_dir; | ||
| use crate::harness::HarnessBundle; | ||
|
|
||
| const MANIFEST_FILE: &str = "sync.json"; |
There was a problem hiding this comment.
We should probably have one place all the CLI configs come from, even if it's hardcoded in some config module somewhere. Then we won't have random filenames strewn throughout the codebase and can potentially make these dynamic in the future if we want.
| /// records (the cache docs themselves survive either way); large | ||
| /// enough that manifest serialization stays noise against the | ||
| /// derives it punctuates. | ||
| const CHECKPOINT_EVERY: usize = 10; |
There was a problem hiding this comment.
Let's make this name more clear alone what it does and throw it at the top of the file (along with any other config variables) so it's consistent where to find them.
| .map(|artifact| (artifact, is_unchanged(records.get(&artifact.id), artifact))) | ||
| .collect(); | ||
| let pending_total = order.iter().filter(|(_, unchanged)| !unchanged).count(); | ||
| let mut progress = Progress::start( |
There was a problem hiding this comment.
This progress should be managed by the binary, not this lib. Something might want to sync, logically, without UI being shown.
| let path = manifest_path()?; | ||
| let dir = path.parent().expect("manifest path has a parent"); | ||
| std::fs::create_dir_all(dir).with_context(|| format!("create {}", dir.display()))?; | ||
| let lock_path = dir.join(format!("{MANIFEST_FILE}.lock")); |
There was a problem hiding this comment.
Same with sync.json above: these names should probably come from some common config location.
# Conflicts: # CLAUDE.md # crates/path-cli/src/artifact.rs
…1 merge toolpath-codex exposes session_id_from_stem (the reader's own filename fallback extraction, moved to paths.rs); sync enumeration and codex import provenance stamp ids through it, replacing the CLI-side codex_artifact_id copy. Sync's progress label and per-type summary use padded_name() over an owned String, and the artifact-type tests merge onto ArtifactType::ALL. Test counts refreshed to measured values.
…st.json The manifest and lock filenames come from the config module (MANIFEST_FILE_NAME / MANIFEST_LOCK_FILE_NAME) instead of a const inside the sync engine, and the on-disk names change from sync.json / sync.json.lock to manifest.json / manifest.json.lock (unreleased, so no migration). The checkpoint interval is renamed MANIFEST_CHECKPOINT_EVERY_WRITES and sits at the top of engine.rs with the other tunables.
2 participants
Second in the stack (on top of the ArtifactType/per-session-import PR). Adds
path p cache sync [types…]: the cache fills itself — sync enumerates sessions across the installed harnesses and derives only what is new or changed, so nobody has top importsession by session.Provider code behind a trait. The provider-specific halves of sync live behind a per-provider
ArtifactSourcetrait (sync/sources.rs: enumerate / stamp / derive, one impl per provider, built from the sameHarnessBundlemanagers the picker uses); the engine (sync/engine.rs) drivesdyn ArtifactSourceand never matches on artifact type. A provider-format change is a change to that provider's source impl, not to central sync logic.Stat-level change detection. Each artifact is enumerated with a fingerprint — source mtime + size for file-backed providers, the DB row's updated-at for opencode/cursor — so deciding "nothing changed" reads no session bodies and a no-op sync is milliseconds. An all-
Nonestamp can never vouch for a record (unknowable freshness re-derives). Claude fingerprints cover the whole rotation chain viaclaude_chain_stamp(max segment mtime + summed sizes): Claude Code rotates files on continuation while the chain keeps its oldest segment's id, so statting the head alone would freeze the fingerprint at the first rotation.The manifest (
~/.toolpath/sync.json) maps artifact type → artifact id →{path?, cache_id?, modified?, size?, synced_at}. Atomic temp+rename0600writes; writers serialize on an advisory lock with locked read-merge-save cycles, so concurrent invocations union their records; checkpointed every 10 writes with derives running newest-first, so an interrupted first sync keeps nearly everything and spent its time on the sessions that matter most. Progress on stderr (live<type> done/totalon a TTY). Sync owns refresh semantics (overwrites what it re-derives); manualp importkeeps error-on-hit. Sessions deleted upstream keep their cache docs and manifest records.Everything that writes the cache records it. Session derives carry a provenance
ArtifactRef(stamped before the source is read), andp import/shareupsert the manifest, so sync never re-derives what they just produced; re-importing a still-fresh artifact is a friendly no-op.p cache rmdowngrades records to known-but-uncached (the next sync re-materializes, and sync verifies doc existence before skipping, so out-of-band deletions self-heal).ArtifactTypegainsGit: recorded on import, never re-derived (repos aren't discoverable).shareuploads straight from the cache when the manifest shows the picked session unchanged (one targeted stat, no sibling enumeration).Tests:
cargo clippy -D warningsclean; 462 tests pass at this point in the stack.Need help on this PR? Tag
/codesmithwith what you need. Autofix is disabled.Stack (review bottom-up): #133 types/imports → #134 cache sync → #135 query auto-sync → #136 parent-dir. Supersedes #131.