feat(cli): p cache sync — incremental ingestion with a manifest artifact index

CHANGELOG.md

@@ -2,6 +2,100 @@
 
 All notable changes to the Toolpath workspace are documented here.
 
+## `path p cache sync` — incremental session ingestion — 2026-07-16
+
+Adds `path p cache sync [types…]`, the first step toward a cache that
+fills itself: it enumerates sessions across the installed agent
+harnesses and derives into the cache only what is new or changed since
+the last sync, so users no longer have to `p import` each session by
+hand.
+
+- **`path-cli`** (0.16.0):
+  - `path p cache sync
+    [claude|gemini|codex|opencode|cursor|pi|copilot|git]…` syncs the
+    named artifact types; with no arguments it syncs every harness.
+    Per-type summary on stderr (`claude   3 new, 1 updated, 240
+    unchanged`); on a default run, harnesses with no sessions stay
+    silent, explicit types always report. Derivation failures warn and
+    tally as `failed` without aborting the run.
+  - The provider-specific halves of sync live behind a per-provider
+    `ArtifactSource` trait (`sync/sources.rs`: enumerate / stamp /
+    derive, one impl per provider) with the engine
+    (`sync/engine.rs`) never matching on artifact type — a provider
+    change is a change to that provider's source, not to the engine.
+    Sources derive through the same provider managers as `p import`
+    (the `derive_*_session_with` helpers), so listing and derivation
+    always agree on provider roots.
+  - The sync manifest at `~/.toolpath/manifest.json` maps artifact type →
+    artifact id → `{path?, cache_id?, modified?, size?, synced_at}`.
+    Change detection is stat-level — source mtime + size for the
+    file-backed providers, the DB row's updated-at for opencode/cursor
+    — so deciding "nothing changed" reads no session bodies and a
+    no-op sync is milliseconds. An all-`None` stamp can never vouch
+    for a record in the unchanged gate — unknowable freshness
+    re-derives. The manifest is written atomically (temp file +
+    rename, `0600`) and checkpointed every 10 writes with derives
+    running newest-first, so an interrupted first run keeps nearly
+    everything it derived — and spent its time on the sessions that
+    matter most. Pending work reports progress on stderr (live
+    `<type> done/total` on a TTY, a plain line every 25 items
+    otherwise; no-op syncs stay silent).
+  - Manifest writers are serialized by an advisory lock
+    (`manifest.json.lock`) and every write is a locked read-merge-save —
+    sync checkpoints merge only the records the run wrote — so
+    concurrent invocations union their records instead of clobbering
+    each other.
+  - Sync owns refresh semantics: it overwrites cache entries it
+    re-derives (no `--force` needed), while manual `p import` keeps its
+    error-on-hit default. Sessions deleted upstream keep both their
+    cache document and their manifest record.
+  - The manifest is an artifact *index*, not just a cache inventory:
+    `cache_id` on a record is optional, and a record without one means
+    "known, not materialized". `p cache rm` downgrades records instead
+    of orphaning them (the next sync re-materializes the doc), and
+    sync verifies doc existence before skipping, so out-of-band
+    deletions self-heal too.
+  - `p import` and `share` record what they write: session derives
+    carry a provenance `ArtifactRef` (source stamped before the read,
+    in `DerivedDoc.provenance`), and the cache write upserts the
+    manifest — so sync never re-derives an artifact that import or
+    share just produced, and re-importing an artifact whose record is
+    still fresh is a friendly no-op instead of an exists-error.
+    `ArtifactType` gains `Git`: git imports are recorded (repo path +
+    `<repo-tag>-<graph-id>` id) but never re-derived by sync, since
+    repos aren't discoverable. Github and pathbase stay out of the
+    manifest — remote services, not artifact sources.
+  - Claude fingerprints cover the whole session chain. Claude Code
+    rotates to a new JSONL file on continuation (plan-mode exit,
+    resume, fork) while `list_conversations` keys the chain by its
+    *oldest* segment — appends land in the newest file, so statting
+    the head file would freeze the fingerprint at the first rotation
+    and sync would never see later turns. All three stamp sites
+    (enumeration, import provenance, the `share` fast path) share
+    `claude_chain_stamp`: max mtime across the chain's segments plus
+    the sum of their sizes — exactly the files `read_conversation`
+    merges, so the fingerprint and the derived doc move in lockstep
+    however rotation behavior shifts across Claude Code versions.
+    Import also normalizes its manifest key to the chain head, so
+    importing a successor-segment id records the artifact sync will
+    look for.
+  - `share` uploads straight from the cache when the picked session is
+    unchanged since its last sync (manifest stamp matches a fresh stat
+    and the doc exists) — re-deriving would reproduce the same bytes.
+    The freshness stat targets that one artifact directly, no sibling
+    enumeration; a grown session steps around the fast path and
+    derives as before.
+  - Copilot participates in sync like every other harness: an
+    `ArtifactType::Copilot` with stat-only enumeration over
+    `session-state/<id>/events.jsonl`, and per-session import loops
+    with provenance.
+- **`toolpath-codex`** (0.6.1, extends the bump below):
+  `session_id_from_stem` is public — the trailing UUID of a rollout
+  filename stem (or the whole stem when it has none), the same
+  extraction the reader uses for `Session.id`'s filename fallback.
+  Codex sync enumeration and import provenance stamp ids through it
+  instead of a CLI-side copy of the filename convention.
+
 ## One artifact-type layer and per-session imports — 2026-07-16
 
 Groundwork for a cache that fills itself: one enum for artifact

README.md

@@ -99,6 +99,10 @@ path p import opencode
 # List what's in the cache
 path p cache ls
 
+# Ingest new/changed agent sessions into the cache (all harnesses, or named ones)
+path p cache sync
+path p cache sync claude codex
+
 # Export a cached document back into a Claude Code session
 path p export claude --input claude-<session-id> --project /path/to/resume
 

crates/path-cli/src/artifact.rs

@@ -1,12 +1,18 @@
 //! [`ArtifactType`], the single enum naming the artifact sources the
-//! CLI operates over.
+//! CLI operates over, plus `ArtifactRef` — an artifact's identity
+//! and stat-level fingerprint — and the stamp helpers that produce
+//! those fingerprints for sync and import provenance.
 
 /// The kind of artifact an operation ranges over. One enum, used
-/// everywhere a command names artifact sources (`share`/`resume`
-/// `--harness` via the [`Harness`](crate::harness::Harness) layer,
-/// import cache-id prefixes); `name()` doubles as the cache-id prefix.
-/// Github and pathbase are absent on purpose: they are remote
-/// services, not local artifact sources.
+/// everywhere a command names artifact sources (`p cache sync` types,
+/// `share`/`resume` `--harness` via the
+/// [`Harness`](crate::harness::Harness) layer, import cache-id
+/// prefixes); `name()` doubles as the manifest key and cache-id
+/// prefix. Git artifacts are recorded in the manifest when imported
+/// but are not *discoverable* — there is no machine-wide registry of
+/// repos to enumerate — so sync never re-derives them. Github and
+/// pathbase are absent on purpose: they are remote services, not
+/// local artifact sources.
 #[derive(Copy, Clone, Debug, PartialEq, Eq, Hash, clap::ValueEnum)]
 #[value(rename_all = "lower")]
 pub enum ArtifactType {
@@ -21,6 +27,18 @@ pub enum ArtifactType {
 }
 
 impl ArtifactType {
+    /// Every artifact type, in presentation order.
+    pub(crate) const ALL: [ArtifactType; 8] = [
+        ArtifactType::Claude,
+        ArtifactType::Gemini,
+        ArtifactType::Codex,
+        ArtifactType::Opencode,
+        ArtifactType::Cursor,
+        ArtifactType::Pi,
+        ArtifactType::Copilot,
+        ArtifactType::Git,
+    ];
+
     pub(crate) fn name(&self) -> &'static str {
         match self {
             ArtifactType::Claude => "claude",
@@ -62,33 +80,95 @@ impl ArtifactType {
     }
 }
 
+/// An artifact's identity plus the stat-level fingerprint of its
+/// source. Sync enumerates these for change detection (producing one
+/// never parses session bodies), and `p import`/`share` fill one as
+/// the provenance of each derived document so the write can be
+/// recorded in the manifest.
+#[derive(Debug, Clone)]
+pub(crate) struct ArtifactRef {
+    pub(crate) artifact_type: ArtifactType,
+    pub(crate) id: String,
+    /// Filesystem path the artifact is keyed under, for path-keyed
+    /// providers (the project directory; the repo for git).
+    pub(crate) path: Option<String>,
+    /// Source mtime (file providers) or updated-at (DB providers).
+    pub(crate) modified: Option<chrono::DateTime<chrono::Utc>>,
+    /// Source file size; `None` for DB-backed providers.
+    pub(crate) size: Option<u64>,
+}
+
+/// (mtime, size) of a file, both `None` when the stat fails.
+pub(crate) fn stat_stamp(
+    path: &std::path::Path,
+) -> (Option<chrono::DateTime<chrono::Utc>>, Option<u64>) {
+    match std::fs::metadata(path) {
+        Ok(md) => (
+            md.modified()
+                .ok()
+                .map(chrono::DateTime::<chrono::Utc>::from),
+            Some(md.len()),
+        ),
+        Err(_) => (None, None),
+    }
+}
+
+/// Stat-level fingerprint of a whole claude session chain: max mtime
+/// across the chain's segment files plus the sum of their sizes. Claude
+/// Code rotates to a new file on continuation (plan-mode exit, resume,
+/// fork) while the chain keeps the *first* segment's id — appends land
+/// in the newest segment, so statting the head file alone would freeze
+/// the fingerprint at the first rotation and sync would never see the
+/// later turns. The chain here is exactly the set of files
+/// `read_conversation` merges, so the fingerprint and the derived doc
+/// move in lockstep. The chain index is already built (and cached) by
+/// the `list_conversations` call every caller makes first.
+pub(crate) fn claude_chain_stamp(
+    mgr: &toolpath_claude::ClaudeConvo,
+    project: &str,
+    session: &str,
+) -> (Option<chrono::DateTime<chrono::Utc>>, Option<u64>) {
+    let segments = match mgr.session_chain(project, session) {
+        Ok(segments) if !segments.is_empty() => segments,
+        _ => vec![session.to_string()],
+    };
+    let mut modified: Option<chrono::DateTime<chrono::Utc>> = None;
+    let mut size: Option<u64> = None;
+    for segment in &segments {
+        let Ok(file) = mgr.resolver().conversation_file(project, segment) else {
+            continue;
+        };
+        let (m, s) = stat_stamp(&file);
+        if let Some(m) = m {
+            modified = Some(modified.map_or(m, |cur| cur.max(m)));
+        }
+        if let Some(s) = s {
+            size = Some(size.unwrap_or(0) + s);
+        }
+    }
+    (modified, size)
+}
+
 #[cfg(test)]
 mod type_tests {
     use super::ArtifactType;
 
-    /// Every artifact type, in presentation order.
-    const ALL: [ArtifactType; 8] = [
-        ArtifactType::Claude,
-        ArtifactType::Gemini,
-        ArtifactType::Codex,
-        ArtifactType::Opencode,
-        ArtifactType::Cursor,
-        ArtifactType::Pi,
-        ArtifactType::Copilot,
-        ArtifactType::Git,
-    ];
-
     #[test]
     fn names_are_distinct() {
-        let names: std::collections::HashSet<&str> = ALL.iter().map(|t| t.name()).collect();
-        assert_eq!(names.len(), ALL.len());
+        let names: std::collections::HashSet<&str> =
+            ArtifactType::ALL.iter().map(|t| t.name()).collect();
+        assert_eq!(names.len(), ArtifactType::ALL.len());
     }
 
     #[test]
     fn name_column_width_is_the_longest_name() {
-        let longest = ALL.iter().map(|t| t.name().len()).max().unwrap();
+        let longest = ArtifactType::ALL
+            .iter()
+            .map(|t| t.name().len())
+            .max()
+            .unwrap();
         assert_eq!(ArtifactType::NAME_COLUMN_WIDTH, longest);
-        for t in ALL {
+        for t in ArtifactType::ALL {
             assert_eq!(t.padded_name().len(), ArtifactType::NAME_COLUMN_WIDTH);
         }
     }
@@ -106,7 +186,7 @@ mod type_tests {
 
     #[test]
     fn parse_roundtrips_every_name() {
-        for t in ALL {
+        for t in ArtifactType::ALL {
             assert_eq!(ArtifactType::parse(t.name()), Some(t));
         }
         assert_eq!(ArtifactType::parse("frobnicate"), None);

crates/path-cli/src/cmd_cache.rs

@@ -15,12 +15,22 @@ pub enum CacheOp {
         /// Cache id (filename without `.json`)
         id: String,
     },
+    /// Ingest agent sessions into the cache, deriving only what is new
+    /// or changed since the last sync (tracked in `$CONFIG_DIR/manifest.json`)
+    #[cfg(not(target_os = "emscripten"))]
+    Sync {
+        /// Artifact types to sync (default: every agent harness)
+        #[arg(value_enum)]
+        types: Vec<crate::artifact::ArtifactType>,
+    },
 }
 
 pub fn run(op: CacheOp) -> Result<()> {
     match op {
         CacheOp::Ls => run_ls(),
         CacheOp::Rm { id } => run_rm(&id),
+        #[cfg(not(target_os = "emscripten"))]
+        CacheOp::Sync { types } => crate::sync::run(types),
     }
 }
 
@@ -38,6 +48,12 @@ fn run_ls() -> Result<()> {
 
 fn run_rm(id: &str) -> Result<()> {
     remove_cached(id)?;
+    // The artifact is still real — downgrade its manifest record to
+    // "known, not cached" so the next sync can re-materialize it.
+    #[cfg(not(target_os = "emscripten"))]
+    if let Err(e) = crate::sync::evict_cache_id(id) {
+        eprintln!("warning: sync manifest not updated: {e}");
+    }
     eprintln!("Removed {id}");
     Ok(())
 }

crates/path-cli/src/cmd_import.rs

@@ -15,7 +15,7 @@ use std::path::PathBuf;
 use toolpath::v1::Graph;
 
 #[cfg(not(target_os = "emscripten"))]
-use crate::artifact::ArtifactType;
+use crate::artifact::{ArtifactRef, ArtifactType};
 #[cfg(not(target_os = "emscripten"))]
 use crate::cache::make_id;
 use crate::cache::write_cached;
@@ -217,8 +217,29 @@ fn emit(docs: &[DerivedDoc], force: bool, no_cache: bool, pretty: bool) -> Resul
             };
             println!("{}", json);
         } else {
+            // `p cache sync` fills the cache under these same ids;
+            // re-importing an artifact whose record is still fresh is
+            // a no-op, not an exists-error.
+            #[cfg(not(target_os = "emscripten"))]
+            if !force
+                && let Some(stub) = &d.provenance
+                && crate::sync::record_is_current(stub, &d.cache_id)
+            {
+                println!("{}", crate::cache::cache_path(&d.cache_id)?.display());
+                eprintln!(
+                    "{} is already up to date (pass --force to re-derive)",
+                    d.cache_id
+                );
+                continue;
+            }
             let path = write_cached(&d.cache_id, &d.doc, force)?;
             println!("{}", path.display());
+            #[cfg(not(target_os = "emscripten"))]
+            if let Some(stub) = &d.provenance
+                && let Err(e) = crate::sync::record_artifact(stub, &d.cache_id)
+            {
+                eprintln!("warning: sync manifest not updated: {e}");
+            }
             let summary = doc_summary(&d.doc);
             eprintln!("Imported {} → {}", summary, d.cache_id);
         }
@@ -324,7 +345,17 @@ fn derive_git(
         let repo_tag = short_path_hash(&canonical.to_string_lossy());
         let inner = doc_inner_id(&doc);
         let cache_id = make_id(ArtifactType::Git.name(), &format!("{repo_tag}-{inner}"));
-        Ok(vec![DerivedDoc { cache_id, doc }])
+        Ok(vec![DerivedDoc {
+            cache_id,
+            doc,
+            provenance: Some(ArtifactRef {
+                artifact_type: ArtifactType::Git,
+                id: format!("{repo_tag}-{inner}"),
+                path: Some(canonical.to_string_lossy().into_owned()),
+                modified: None,
+                size: None,
+            }),
+        }])
     }
 }
 
@@ -381,7 +412,11 @@ fn derive_github(
         let path = toolpath_github::derive_pull_request(&owner, &repo_name, pr_number, &config)?;
         let doc = Graph::from_path(path);
         let cache_id = make_id("github", &format!("{owner}_{repo_name}-{pr_number}"));
-        Ok(vec![DerivedDoc { cache_id, doc }])
+        Ok(vec![DerivedDoc {
+            cache_id,
+            doc,
+            provenance: None,
+        }])
     }
 }