diff --git a/CHANGELOG.md b/CHANGELOG.md index 13265c85..4c3aceb3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,100 @@ All notable changes to the Toolpath workspace are documented here. +## `path p cache sync` — incremental session ingestion — 2026-07-16 + +Adds `path p cache sync [types…]`, the first step toward a cache that +fills itself: it enumerates sessions across the installed agent +harnesses and derives into the cache only what is new or changed since +the last sync, so users no longer have to `p import` each session by +hand. + +- **`path-cli`** (0.16.0): + - `path p cache sync + [claude|gemini|codex|opencode|cursor|pi|copilot|git]…` syncs the + named artifact types; with no arguments it syncs every harness. + Per-type summary on stderr (`claude 3 new, 1 updated, 240 + unchanged`); on a default run, harnesses with no sessions stay + silent, explicit types always report. Derivation failures warn and + tally as `failed` without aborting the run. + - The provider-specific halves of sync live behind a per-provider + `ArtifactSource` trait (`sync/sources.rs`: enumerate / stamp / + derive, one impl per provider) with the engine + (`sync/engine.rs`) never matching on artifact type — a provider + change is a change to that provider's source, not to the engine. + Sources derive through the same provider managers as `p import` + (the `derive_*_session_with` helpers), so listing and derivation + always agree on provider roots. + - The sync manifest at `~/.toolpath/manifest.json` maps artifact type → + artifact id → `{path?, cache_id?, modified?, size?, synced_at}`. + Change detection is stat-level — source mtime + size for the + file-backed providers, the DB row's updated-at for opencode/cursor + — so deciding "nothing changed" reads no session bodies and a + no-op sync is milliseconds. An all-`None` stamp can never vouch + for a record in the unchanged gate — unknowable freshness + re-derives. The manifest is written atomically (temp file + + rename, `0600`) and checkpointed every 10 writes with derives + running newest-first, so an interrupted first run keeps nearly + everything it derived — and spent its time on the sessions that + matter most. Pending work reports progress on stderr (live + ` done/total` on a TTY, a plain line every 25 items + otherwise; no-op syncs stay silent). + - Manifest writers are serialized by an advisory lock + (`manifest.json.lock`) and every write is a locked read-merge-save — + sync checkpoints merge only the records the run wrote — so + concurrent invocations union their records instead of clobbering + each other. + - Sync owns refresh semantics: it overwrites cache entries it + re-derives (no `--force` needed), while manual `p import` keeps its + error-on-hit default. Sessions deleted upstream keep both their + cache document and their manifest record. + - The manifest is an artifact *index*, not just a cache inventory: + `cache_id` on a record is optional, and a record without one means + "known, not materialized". `p cache rm` downgrades records instead + of orphaning them (the next sync re-materializes the doc), and + sync verifies doc existence before skipping, so out-of-band + deletions self-heal too. + - `p import` and `share` record what they write: session derives + carry a provenance `ArtifactRef` (source stamped before the read, + in `DerivedDoc.provenance`), and the cache write upserts the + manifest — so sync never re-derives an artifact that import or + share just produced, and re-importing an artifact whose record is + still fresh is a friendly no-op instead of an exists-error. + `ArtifactType` gains `Git`: git imports are recorded (repo path + + `-` id) but never re-derived by sync, since + repos aren't discoverable. Github and pathbase stay out of the + manifest — remote services, not artifact sources. + - Claude fingerprints cover the whole session chain. Claude Code + rotates to a new JSONL file on continuation (plan-mode exit, + resume, fork) while `list_conversations` keys the chain by its + *oldest* segment — appends land in the newest file, so statting + the head file would freeze the fingerprint at the first rotation + and sync would never see later turns. All three stamp sites + (enumeration, import provenance, the `share` fast path) share + `claude_chain_stamp`: max mtime across the chain's segments plus + the sum of their sizes — exactly the files `read_conversation` + merges, so the fingerprint and the derived doc move in lockstep + however rotation behavior shifts across Claude Code versions. + Import also normalizes its manifest key to the chain head, so + importing a successor-segment id records the artifact sync will + look for. + - `share` uploads straight from the cache when the picked session is + unchanged since its last sync (manifest stamp matches a fresh stat + and the doc exists) — re-deriving would reproduce the same bytes. + The freshness stat targets that one artifact directly, no sibling + enumeration; a grown session steps around the fast path and + derives as before. + - Copilot participates in sync like every other harness: an + `ArtifactType::Copilot` with stat-only enumeration over + `session-state//events.jsonl`, and per-session import loops + with provenance. +- **`toolpath-codex`** (0.6.1, extends the bump below): + `session_id_from_stem` is public — the trailing UUID of a rollout + filename stem (or the whole stem when it has none), the same + extraction the reader uses for `Session.id`'s filename fallback. + Codex sync enumeration and import provenance stamp ids through it + instead of a CLI-side copy of the filename convention. + ## One artifact-type layer and per-session imports — 2026-07-16 Groundwork for a cache that fills itself: one enum for artifact diff --git a/CLAUDE.md b/CLAUDE.md index d75497af..2aca51c7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -129,6 +129,8 @@ cargo run -p path-cli -- p export pathbase --input # Plumbing: manage the cache cargo run -p path-cli -- p cache ls cargo run -p path-cli -- p cache rm +cargo run -p path-cli -- p cache sync # ingest new/changed sessions from every harness +cargo run -p path-cli -- p cache sync claude codex # only these artifact types # Inspect / analyze cargo run -p path-cli -- p render dot --input doc.json @@ -165,7 +167,7 @@ cargo run -p path-cli -- auth logout and no deprecation shim. They all now live exclusively under `path p …`. -The **cache** at `~/.toolpath/documents/.json` is the single landing zone for every `import` (and for `import pathbase` downloads). Cache id is `-` — e.g. `claude-abc123`, `git-main`, `pathbase-alex-pathstash-path-pr-42` (Pathbase paths key on `--`, anon paths on `anon-pathstash-`). Files are `0600`, parent directory `0700`. `$TOOLPATH_CONFIG_DIR` overrides the root. Default behavior: error on cache hit; pass `--force` to overwrite. `--no-cache` sends the JSON to stdout for shell composition. +The **cache** at `~/.toolpath/documents/.json` is the single landing zone for every `import` (and for `import pathbase` downloads). Cache id is `-` — e.g. `claude-abc123`, `git-main`, `pathbase-alex-pathstash-path-pr-42` (Pathbase paths key on `--`, anon paths on `anon-pathstash-`). Files are `0600`, parent directory `0700`. `$TOOLPATH_CONFIG_DIR` overrides the root. Default behavior: error on cache hit; pass `--force` to overwrite. `--no-cache` sends the JSON to stdout for shell composition. `p cache sync` fills the cache incrementally from the installed agent harnesses (see "Things to know") and always overwrites what it re-derives. `path auth login` prints `/auth/cli`; the user opens it, logs in, and pastes the 8-character code back into the CLI. The CLI calls @@ -200,13 +202,13 @@ Tests live alongside the code (`#[cfg(test)] mod tests`), plus `path-cli` has in - `toolpath-github`: 32 unit + 3 doc tests (mapping, DAG construction, fixtures) - `toolpath-claude`: 229 unit + 18 integration + 6 doc tests (path resolution, conversation reading, query, chaining, watcher, derive, metadata first-user-message, group_id grouping + once-per-message usage totals) - `toolpath-gemini`: 165 unit + 29 integration + 5 doc tests (path resolution, chat-file parsing, query, watcher, derive, provider, round-trip fidelity, thoughts-folded-into-output + reasoning breakdown round-trip) -- `toolpath-codex`: 82 unit + 51 integration + 2 doc tests (rollout parsing, provider assembly, patch-fidelity derive, real-session fixture, source→path fidelity invariants, JSON wire-level round-trip, per-turn token deltas from cumulative counters, reasoning breakdown) +- `toolpath-codex`: 83 unit + 51 integration + 2 doc tests (rollout parsing, provider assembly, patch-fidelity derive, real-session fixture, source→path fidelity invariants, JSON wire-level round-trip, per-turn token deltas from cumulative counters, reasoning breakdown) - `toolpath-copilot`: 63 unit + 8 integration + 1 doc test (`events.jsonl` envelope/event-type classification incl. `session.start` nested `context` + `tool.execution` `result.content` + `assistant.message` `reasoningText`/`outputTokens`, `session.shutdown` `tokenDetails`, path resolution incl. legacy `history-session-state/`, reader malformed-line tolerance without env races, tolerant `workspace.yaml` parse, `to_view` turn/tool/delegation assembly + per-turn token usage + shutdown-total merge, id-based **and** id-less positional tool pairing, position-stable turn ids, native file-state diff from `result.detailedContent`, `CopilotProjector` round-trip + foreign-tool-name remap). Ships a **real captured feature-elicit session** at `tests/fixtures/real-session.jsonl` (also `test-fixtures/copilot/convo.jsonl`) driving `real_fixture_roundtrip.rs` (forward invariants, projection round-trip fidelity, wire-level serde value-identity). The projector is exercised by the cross-harness matrix in `path-cli`. - `toolpath-opencode`: 52 unit + 19 integration + 1 doc test (SQLite reader, JSON payload serde, provider assembly, snapshot-based derive, tool-input fallback for gitignored paths, reasoning breakdown) - `toolpath-cursor`: 78 unit + 8 integration round-trip + 1 real-DB sanity + 1 doc test (state.vscdb SQLite reader, bubble store + composer header parsing, content-addressed blob lookup, projector with full TOOL_TABLE coverage, JSONL transcript ingest in `examples/dump_fixture.rs`) - `toolpath-pi`: 133 unit + 26 integration + 5 doc tests (types, paths, error, reader, io, provider) - `toolpath-dot`: 30 unit + 2 doc tests (render, visual conventions, escaping) -- `path-cli`: 327 unit + 107 integration tests (import/export/cache, track sessions, merge, validate, roundtrip, render-md snapshots, deprecation aliases, pathbase HTTP mock-server tests, fzf-friendly TSV output, `path resume` orchestration with injectable `ExecStrategy`, `path query`/`path kind` jaq filters + kind-selector matching + step wrapping over a `$TOOLPATH_CONFIG_DIR` cache sandbox, streaming-planner recognition + streamed-output-equals-slurp equality checks, and per-session import loops — maximal thinking ingest, unreadable-session skipping — over resolver-injected provider fixtures). For an end-to-end check against a real Pathbase deployment, run `scripts/test-pathbase-live.sh ` — it does an anon round-trip in a sandboxed config dir and, if you're logged into that URL, an authed pathstash round-trip too. +- `path-cli`: 348 unit + 115 integration tests (import/export/cache, track sessions, merge, validate, roundtrip, render-md snapshots, deprecation aliases, pathbase HTTP mock-server tests, fzf-friendly TSV output, `path resume` orchestration with injectable `ExecStrategy`, `path query`/`path kind` jaq filters + kind-selector matching + step wrapping over a `$TOOLPATH_CONFIG_DIR` cache sandbox, streaming-planner recognition + streamed-output-equals-slurp equality checks, `p cache sync` incremental ingestion — stat fingerprints, refresh-overwrite, per-type filtering, failure tallying — over resolver-injected provider fixtures, and import/share manifest recording end-to-end). For an end-to-end check against a real Pathbase deployment, run `scripts/test-pathbase-live.sh ` — it does an anon round-trip in a sandboxed config dir and, if you're logged into that URL, an authed pathstash round-trip too. - `toolpath-cli`: 0 tests (it's a one-line `path_cli::run()` shim crate that exists only so `cargo install toolpath-cli` keeps installing the `path` binary) Validate example documents: `for f in examples/*.json; do cargo run -p path-cli -- p validate --input "$f"; done` @@ -273,7 +275,8 @@ Build the site after changes: `cd site && pnpm run build` (should produce 11 pag - Format references for the agent on-disk formats we derive from live at `docs/agents/formats/`. The Claude Code format (`~/.claude/projects/…` JSONL) gets the deepest treatment — twelve focused docs at `docs/agents/formats/claude-code/` covering envelope, entry types, tools, session chains, compaction, writing-compatible JSONL, a linear walkthrough, and a version-keyed changelog. Sibling single-file references: `codex.md`, `gemini.md`, `opencode.md`. Keep them in sync with their derive crates when fields or behaviors change. - Interactive session selection: `path p import ` (claude / gemini / pi / codex / opencode) auto-launches a fuzzy picker when stdin and stderr are TTYs and no `--session` was given. Backend: external `fzf` if on `$PATH`, otherwise the embedded skim picker (default-feature `embedded-picker`, defined in `crates/path-cli/src/skim_picker.rs`). Multi-select (TAB) produces a `Graph` document; single-select produces a `Path`. The picker uses `path show --…` as its `--preview` command. When neither backend can run (no TTY, or `--no-default-features` AND no `fzf`), it falls back to most-recent (with `--project`) or prints the manual recipe (without). `path p list --format tsv` is the documented machine-readable surface — column 1 is the project (for claude/gemini/pi) or session id (for codex/opencode), and the trailing column carries `first_user_message` so consumers can fuzzy-match by topic. - Conversation metadata title field: `toolpath-claude::ConversationMetadata`, `toolpath-gemini::ConversationMetadata`, and `toolpath-pi::SessionMeta` all expose `first_user_message: Option` — the first non-empty user-prompt text. Populated cheaply during the metadata pass (single-pass for Claude/Gemini; one extra short read for Pi). Used by the picker UI but useful for any "list sessions by topic" surface. -- `path share` is the one-shot equivalent of `path p import | path p export pathbase`. It probes installed agent harnesses (claude/gemini/codex/opencode/pi), aggregates their sessions into a single fzf picker, and ranks rows whose project (claude/gemini/pi) or recorded cwd (codex/opencode) canonicalizes to the current directory at the top. `--harness` narrows the picker to one provider; `--harness X --session Y` (and `--project P` for keyed providers) skips the picker entirely. Pathbase flags (`--url`, `--anon`, `--repo`, `--slug`, `--public`) match `path export pathbase`. By default the derived doc is written to the cache like `import` does; pass `--no-cache` to skip. The cache ingests maximally (thinking always included), and uploads carry the same full derivation as local projection (`resume`, `p export `) — there is no egress stripping. +- `path share` is the one-shot equivalent of `path p import | path p export pathbase`. It probes installed agent harnesses (claude/gemini/codex/opencode/pi), aggregates their sessions into a single fzf picker, and ranks rows whose project (claude/gemini/pi) or recorded cwd (codex/opencode) canonicalizes to the current directory at the top. `--harness` narrows the picker to one provider; `--harness X --session Y` (and `--project P` for keyed providers) skips the picker entirely. Pathbase flags (`--url`, `--anon`, `--repo`, `--slug`, `--public`) match `path export pathbase`. By default the derived doc is written to the cache like `import` does; pass `--no-cache` to skip. When the manifest shows the picked session unchanged since its last sync (`sync::fresh_cache_id`: stamps match, doc present; the freshness stat targets that one artifact directly, no sibling enumeration), share uploads the cached doc directly instead of re-deriving. The cache ingests maximally (thinking always included), and uploads carry the same full derivation as local projection (`resume`, `p export `) — there is no egress stripping. - `path resume ` is the inverse of `path share`. It accepts a Pathbase URL, an `owner/repo/slug` shorthand, a local toolpath JSON file, or a cache id; resolves it (caching URL fetches under `~/.toolpath/documents/` unless `--no-cache`); validates that the document is a single agent-bearing `Path`; then opens an `fzf` harness picker (skipped with `--harness X`). The picker pre-selects the source harness inferred from `path.meta.source` (`claude-code`/`gemini-cli`/`codex`/`opencode`/`pi`) when it's installed. After picking, `path resume` projects the session into the harness's on-disk layout under the chosen working directory (default: shell cwd; override with `-C, --cwd P`) and `execvp`'s the harness's resume command (`claude -r ` / `gemini --resume ` / `codex resume ` / `opencode --session ` / `pi --session `). On Windows it spawns and waits, propagating the exit code. The exec is mockable via `cmd_resume::ExecStrategy` — production uses `RealExec`; integration tests use `RecordingExec` to capture the recipe without launching a real harness. - `path query` does not load the whole cache into memory when it can avoid it. `crates/path-cli/src/query/plan.rs` parses the jaq filter into jaq's own AST (`jaq_core::load::parse::Term`) and classifies it into a `Plan`: `PerFileStream` (`.[] | g` element-wise work — run per document, print as you go), `Decompose { reduce }` (algebraic aggregations — run the whole filter per file, concatenate the per-file outputs, then run a derived combine: `map`→`add` (array concat), top-N `sort_by(k)|.[:N]`→`add | sort_by(k)|.[:N]`, `length`→`add` over exact integer counts), or `Slurp` (the always-correct whole-array fallback). Recognition is conservative — a non-distributive prefix like `unique`/`group_by` slurps, and so do scalar `add` (float sums re-associate across per-file partials), `min`/`max` (`[] | min == null` poisons the merge), and any unrecognized tail — so **the planner never changes an answer** — `crates/path-cli/src/query/filter.rs` tests assert streamed output equals slurp byte-for-byte. `filter::execute` compiles the filter once (jaq's compiled `Filter` is fully owned, so it's reused across files) and drives the plan; `mod.rs::stream_files` yields one document's wrapped steps at a time. `TOOLPATH_QUERY_EXPLAIN=1` prints the chosen plan to stderr. No user-facing flag — it's automatic. Tie-break caveat: a streamed top-N matches slurp's *ranking*, but boundary ties may resolve to different specific rows. -- `ArtifactType` (`crates/path-cli/src/artifact.rs`) is the general enum naming artifact sources — the seven agent harnesses (incl. copilot) plus `Git` (8 variants). Github and pathbase are deliberately not artifact types: they are remote services, not local artifact sources. It derives `clap::ValueEnum` and is used by `ArtifactRow.artifact_type` and `cmd_import`'s cache-id prefixes (`name()` is the `make_id` source string). The deliberately parallel `Harness` enum (`crates/path-cli/src/harness.rs`, alongside `HarnessBundle`) names the seven agent *runtimes* — things sessions can be shared from and resumed into — and is what `share`/`resume` `--harness` take, so future non-harness artifact types stay unrepresentable there (you can't resume into a git repo). `Harness::artifact_type()` maps into the general enum; `ArtifactType::harness()` is the partial inverse. Keep new code on `ArtifactType` unless it's genuinely harness-only. +- Cache sync: `path p cache sync [types…]` (`crates/path-cli/src/artifact.rs`: `ArtifactType` + `ArtifactRef` + the stamp helpers; `sync/engine.rs`: manifest + ingestion loop; `sync/sources.rs`: an `ArtifactSource` trait — enumerate / stamp / derive — with one impl per provider, so the engine never matches on artifact type) incrementally ingests artifacts into the cache — no args syncs every artifact type. Change detection is **stat-level**: each artifact is enumerated as an `ArtifactRef` whose fingerprint is the source file's mtime + size (claude: the *whole session chain* — max segment mtime + summed segment sizes via `claude_chain_stamp`, because Claude Code rotates to a new file on continuation while the chain keeps its oldest segment's id, so appends land in the newest file, not the head; the chain comes from the same cached index `list_conversations` builds; codex: rollout file, id from the stem's trailing UUID; pi: session file, id from a one-line header peek; copilot: `session-state//events.jsonl`, pure read-dir + stat) or the DB row's updated-at (opencode: header-only `SELECT time_updated`; cursor: composer headers' `lastUpdatedAt`, bubble-less drafts skipped, workspace-less composers *included* unlike `share`). Gemini enumerates via `PathResolver::list_session_entries` (`toolpath-gemini` 0.6.1), whose identity peek is bounded to the first 4 KiB of a main file. Deciding "nothing changed" reads no session bodies — a no-op sync is milliseconds. Changed/new artifacts derive through the same provider managers (each source calls the `derive_*_session_with` helpers in `derive.rs`). Manifest at `~/.toolpath/manifest.json`: artifact type → artifact id → `{path?, cache_id, modified?, size?, synced_at}`; atomic temp+rename writes, `0600`, checkpointed every 10 writes (interruption-safe: a killed run keeps nearly everything it derived, and derives run newest-first so partial progress covers the sessions that matter most); writers serialize on an advisory lock (`manifest.json.lock`) and every write is a locked read-merge-save — checkpoints merge only the records the run wrote — so concurrent invocations (imports, parallel syncs) union their records instead of clobbering each other. Pending work reports progress on stderr (`\r`-updating ` done/total` on a TTY, a plain line every 25 items otherwise; no-op syncs stay silent). Sync always writes the cache with force — refresh semantics — and never deletes: artifacts removed upstream keep their cache docs and manifest records (archive, not mirror). Derivation failures warn and tally, they don't abort. A record's `cache_id` is *optional*: a record without one is "known, not materialized" — created when `p cache rm` evicts a doc (rm downgrades the record; the next sync re-materializes it, and sync also verifies the doc file actually exists before skipping, so even out-of-band deletions self-heal). Claude derives leave `DeriveConfig.project_path` unset so `path.base` comes from the session's own recorded cwd rather than the lossy slug. `p import` and `share` record what they write: every session derive carries a provenance `ArtifactRef` (stamped *before* the source is read, in `DerivedDoc.provenance`), and the cache-write sites call `sync::record_artifact` so the next sync sees those artifacts as unchanged instead of re-deriving them. Every import flow — explicit `--session`, picker multi-select, `--all`, and the most-recent fallbacks — loops the per-session helpers, so every session write is recorded; there is no bulk `derive_project` path in the CLI anymore, and `p import pi --all` now emits one Path per session like every other provider (it used to emit a single combined Graph). `--no-cache` paths record nothing: the manifest describes the cache. +- `ArtifactType` (`crates/path-cli/src/artifact.rs`) is the general enum naming artifact sources — the seven agent harnesses (incl. copilot) plus `Git` (8 variants). Git artifacts are *recorded* in the manifest by `p import git` (id `-`, `path` = the repo directory) but never *discovered* — there is no machine-wide registry of repos — so sync reports them and leaves them alone. Github and pathbase are deliberately not artifact types: they are remote services, not local artifact sources, and their imports stay out of the manifest. It derives `clap::ValueEnum` and is used by `p cache sync` types, the sync manifest keys, `ArtifactRow.artifact_type`, and `cmd_import`'s cache-id prefixes (`name()` is both the manifest key and the `make_id` source string). The deliberately parallel `Harness` enum (`crates/path-cli/src/harness.rs`, alongside `HarnessBundle`) names the seven agent *runtimes* — things sessions can be shared from and resumed into — and is what `share`/`resume` `--harness` take, so future non-harness artifact types stay unrepresentable there (you can't resume into a git repo). `Harness::artifact_type()` maps into the general enum; `ArtifactType::harness()` is the partial inverse. Keep new code on `ArtifactType` unless it's genuinely harness-only. diff --git a/README.md b/README.md index 5d481f03..f587db82 100644 --- a/README.md +++ b/README.md @@ -99,6 +99,10 @@ path p import opencode # List what's in the cache path p cache ls +# Ingest new/changed agent sessions into the cache (all harnesses, or named ones) +path p cache sync +path p cache sync claude codex + # Export a cached document back into a Claude Code session path p export claude --input claude- --project /path/to/resume diff --git a/crates/path-cli/src/artifact.rs b/crates/path-cli/src/artifact.rs index 367f6ea2..b7127584 100644 --- a/crates/path-cli/src/artifact.rs +++ b/crates/path-cli/src/artifact.rs @@ -1,12 +1,18 @@ //! [`ArtifactType`], the single enum naming the artifact sources the -//! CLI operates over. +//! CLI operates over, plus `ArtifactRef` — an artifact's identity +//! and stat-level fingerprint — and the stamp helpers that produce +//! those fingerprints for sync and import provenance. /// The kind of artifact an operation ranges over. One enum, used -/// everywhere a command names artifact sources (`share`/`resume` -/// `--harness` via the [`Harness`](crate::harness::Harness) layer, -/// import cache-id prefixes); `name()` doubles as the cache-id prefix. -/// Github and pathbase are absent on purpose: they are remote -/// services, not local artifact sources. +/// everywhere a command names artifact sources (`p cache sync` types, +/// `share`/`resume` `--harness` via the +/// [`Harness`](crate::harness::Harness) layer, import cache-id +/// prefixes); `name()` doubles as the manifest key and cache-id +/// prefix. Git artifacts are recorded in the manifest when imported +/// but are not *discoverable* — there is no machine-wide registry of +/// repos to enumerate — so sync never re-derives them. Github and +/// pathbase are absent on purpose: they are remote services, not +/// local artifact sources. #[derive(Copy, Clone, Debug, PartialEq, Eq, Hash, clap::ValueEnum)] #[value(rename_all = "lower")] pub enum ArtifactType { @@ -21,6 +27,18 @@ pub enum ArtifactType { } impl ArtifactType { + /// Every artifact type, in presentation order. + pub(crate) const ALL: [ArtifactType; 8] = [ + ArtifactType::Claude, + ArtifactType::Gemini, + ArtifactType::Codex, + ArtifactType::Opencode, + ArtifactType::Cursor, + ArtifactType::Pi, + ArtifactType::Copilot, + ArtifactType::Git, + ]; + pub(crate) fn name(&self) -> &'static str { match self { ArtifactType::Claude => "claude", @@ -62,33 +80,95 @@ impl ArtifactType { } } +/// An artifact's identity plus the stat-level fingerprint of its +/// source. Sync enumerates these for change detection (producing one +/// never parses session bodies), and `p import`/`share` fill one as +/// the provenance of each derived document so the write can be +/// recorded in the manifest. +#[derive(Debug, Clone)] +pub(crate) struct ArtifactRef { + pub(crate) artifact_type: ArtifactType, + pub(crate) id: String, + /// Filesystem path the artifact is keyed under, for path-keyed + /// providers (the project directory; the repo for git). + pub(crate) path: Option, + /// Source mtime (file providers) or updated-at (DB providers). + pub(crate) modified: Option>, + /// Source file size; `None` for DB-backed providers. + pub(crate) size: Option, +} + +/// (mtime, size) of a file, both `None` when the stat fails. +pub(crate) fn stat_stamp( + path: &std::path::Path, +) -> (Option>, Option) { + match std::fs::metadata(path) { + Ok(md) => ( + md.modified() + .ok() + .map(chrono::DateTime::::from), + Some(md.len()), + ), + Err(_) => (None, None), + } +} + +/// Stat-level fingerprint of a whole claude session chain: max mtime +/// across the chain's segment files plus the sum of their sizes. Claude +/// Code rotates to a new file on continuation (plan-mode exit, resume, +/// fork) while the chain keeps the *first* segment's id — appends land +/// in the newest segment, so statting the head file alone would freeze +/// the fingerprint at the first rotation and sync would never see the +/// later turns. The chain here is exactly the set of files +/// `read_conversation` merges, so the fingerprint and the derived doc +/// move in lockstep. The chain index is already built (and cached) by +/// the `list_conversations` call every caller makes first. +pub(crate) fn claude_chain_stamp( + mgr: &toolpath_claude::ClaudeConvo, + project: &str, + session: &str, +) -> (Option>, Option) { + let segments = match mgr.session_chain(project, session) { + Ok(segments) if !segments.is_empty() => segments, + _ => vec![session.to_string()], + }; + let mut modified: Option> = None; + let mut size: Option = None; + for segment in &segments { + let Ok(file) = mgr.resolver().conversation_file(project, segment) else { + continue; + }; + let (m, s) = stat_stamp(&file); + if let Some(m) = m { + modified = Some(modified.map_or(m, |cur| cur.max(m))); + } + if let Some(s) = s { + size = Some(size.unwrap_or(0) + s); + } + } + (modified, size) +} + #[cfg(test)] mod type_tests { use super::ArtifactType; - /// Every artifact type, in presentation order. - const ALL: [ArtifactType; 8] = [ - ArtifactType::Claude, - ArtifactType::Gemini, - ArtifactType::Codex, - ArtifactType::Opencode, - ArtifactType::Cursor, - ArtifactType::Pi, - ArtifactType::Copilot, - ArtifactType::Git, - ]; - #[test] fn names_are_distinct() { - let names: std::collections::HashSet<&str> = ALL.iter().map(|t| t.name()).collect(); - assert_eq!(names.len(), ALL.len()); + let names: std::collections::HashSet<&str> = + ArtifactType::ALL.iter().map(|t| t.name()).collect(); + assert_eq!(names.len(), ArtifactType::ALL.len()); } #[test] fn name_column_width_is_the_longest_name() { - let longest = ALL.iter().map(|t| t.name().len()).max().unwrap(); + let longest = ArtifactType::ALL + .iter() + .map(|t| t.name().len()) + .max() + .unwrap(); assert_eq!(ArtifactType::NAME_COLUMN_WIDTH, longest); - for t in ALL { + for t in ArtifactType::ALL { assert_eq!(t.padded_name().len(), ArtifactType::NAME_COLUMN_WIDTH); } } @@ -106,7 +186,7 @@ mod type_tests { #[test] fn parse_roundtrips_every_name() { - for t in ALL { + for t in ArtifactType::ALL { assert_eq!(ArtifactType::parse(t.name()), Some(t)); } assert_eq!(ArtifactType::parse("frobnicate"), None); diff --git a/crates/path-cli/src/cmd_cache.rs b/crates/path-cli/src/cmd_cache.rs index 1c5226e4..dd86008a 100644 --- a/crates/path-cli/src/cmd_cache.rs +++ b/crates/path-cli/src/cmd_cache.rs @@ -15,12 +15,22 @@ pub enum CacheOp { /// Cache id (filename without `.json`) id: String, }, + /// Ingest agent sessions into the cache, deriving only what is new + /// or changed since the last sync (tracked in `$CONFIG_DIR/manifest.json`) + #[cfg(not(target_os = "emscripten"))] + Sync { + /// Artifact types to sync (default: every agent harness) + #[arg(value_enum)] + types: Vec, + }, } pub fn run(op: CacheOp) -> Result<()> { match op { CacheOp::Ls => run_ls(), CacheOp::Rm { id } => run_rm(&id), + #[cfg(not(target_os = "emscripten"))] + CacheOp::Sync { types } => crate::sync::run(types), } } @@ -38,6 +48,12 @@ fn run_ls() -> Result<()> { fn run_rm(id: &str) -> Result<()> { remove_cached(id)?; + // The artifact is still real — downgrade its manifest record to + // "known, not cached" so the next sync can re-materialize it. + #[cfg(not(target_os = "emscripten"))] + if let Err(e) = crate::sync::evict_cache_id(id) { + eprintln!("warning: sync manifest not updated: {e}"); + } eprintln!("Removed {id}"); Ok(()) } diff --git a/crates/path-cli/src/cmd_import.rs b/crates/path-cli/src/cmd_import.rs index 0c952340..1d8287b7 100644 --- a/crates/path-cli/src/cmd_import.rs +++ b/crates/path-cli/src/cmd_import.rs @@ -15,7 +15,7 @@ use std::path::PathBuf; use toolpath::v1::Graph; #[cfg(not(target_os = "emscripten"))] -use crate::artifact::ArtifactType; +use crate::artifact::{ArtifactRef, ArtifactType}; #[cfg(not(target_os = "emscripten"))] use crate::cache::make_id; use crate::cache::write_cached; @@ -217,8 +217,29 @@ fn emit(docs: &[DerivedDoc], force: bool, no_cache: bool, pretty: bool) -> Resul }; println!("{}", json); } else { + // `p cache sync` fills the cache under these same ids; + // re-importing an artifact whose record is still fresh is + // a no-op, not an exists-error. + #[cfg(not(target_os = "emscripten"))] + if !force + && let Some(stub) = &d.provenance + && crate::sync::record_is_current(stub, &d.cache_id) + { + println!("{}", crate::cache::cache_path(&d.cache_id)?.display()); + eprintln!( + "{} is already up to date (pass --force to re-derive)", + d.cache_id + ); + continue; + } let path = write_cached(&d.cache_id, &d.doc, force)?; println!("{}", path.display()); + #[cfg(not(target_os = "emscripten"))] + if let Some(stub) = &d.provenance + && let Err(e) = crate::sync::record_artifact(stub, &d.cache_id) + { + eprintln!("warning: sync manifest not updated: {e}"); + } let summary = doc_summary(&d.doc); eprintln!("Imported {} → {}", summary, d.cache_id); } @@ -324,7 +345,17 @@ fn derive_git( let repo_tag = short_path_hash(&canonical.to_string_lossy()); let inner = doc_inner_id(&doc); let cache_id = make_id(ArtifactType::Git.name(), &format!("{repo_tag}-{inner}")); - Ok(vec![DerivedDoc { cache_id, doc }]) + Ok(vec![DerivedDoc { + cache_id, + doc, + provenance: Some(ArtifactRef { + artifact_type: ArtifactType::Git, + id: format!("{repo_tag}-{inner}"), + path: Some(canonical.to_string_lossy().into_owned()), + modified: None, + size: None, + }), + }]) } } @@ -381,7 +412,11 @@ fn derive_github( let path = toolpath_github::derive_pull_request(&owner, &repo_name, pr_number, &config)?; let doc = Graph::from_path(path); let cache_id = make_id("github", &format!("{owner}_{repo_name}-{pr_number}")); - Ok(vec![DerivedDoc { cache_id, doc }]) + Ok(vec![DerivedDoc { + cache_id, + doc, + provenance: None, + }]) } } diff --git a/crates/path-cli/src/cmd_share.rs b/crates/path-cli/src/cmd_share.rs index b3965e3c..d85b780f 100644 --- a/crates/path-cli/src/cmd_share.rs +++ b/crates/path-cli/src/cmd_share.rs @@ -3,7 +3,7 @@ #![cfg(not(target_os = "emscripten"))] -use anyhow::Result; +use anyhow::{Context, Result}; use chrono::{DateTime, Utc}; use clap::Args; use std::path::PathBuf; @@ -801,6 +801,35 @@ fn share_explicit( (false, _) => None, }; + // Fast path: when the manifest shows this exact source state is + // already in the cache, upload the cached doc instead of re-deriving + // — a derive would reproduce it byte-for-byte anyway. + if !args.no_cache + && let Some(cache_id) = crate::sync::fresh_cache_id( + &HarnessBundle::from_environment(), + harness, + project.as_deref(), + session, + ) + { + let doc_path = crate::cache::cache_path(&cache_id)?; + let body = std::fs::read_to_string(&doc_path) + .with_context(|| format!("Failed to read {}", doc_path.display()))?; + eprintln!( + "Cache is current for {} session {cache_id}; uploading without re-deriving", + harness.name() + ); + let summary = format!("{} session {}", harness.name(), cache_id); + let upload = crate::cmd_export::PathbaseUploadArgs { + url: args.url.clone(), + anon: args.anon, + repo: args.repo.clone(), + name: args.name.clone(), + public: args.public, + }; + return crate::cmd_export::run_pathbase_inner(auth, base_url, upload, &body, &summary); + } + let derived = derive_session(harness, project.as_deref(), session)?; let summary = format!("{} session {}", harness.name(), derived.cache_id); @@ -813,6 +842,11 @@ fn share_explicit( // overwrite so cache and upload agree (use `--no-cache` to skip // the cache write entirely). let path = crate::cache::write_cached(&derived.cache_id, &derived.doc, true)?; + if let Some(stub) = &derived.provenance + && let Err(e) = crate::sync::record_artifact(stub, &derived.cache_id) + { + eprintln!("warning: sync manifest not updated: {e}"); + } eprintln!( "Cached {} session → {} ({})", harness.name(), diff --git a/crates/path-cli/src/config.rs b/crates/path-cli/src/config.rs index f9fca692..86a923f8 100644 --- a/crates/path-cli/src/config.rs +++ b/crates/path-cli/src/config.rs @@ -11,6 +11,13 @@ use std::path::PathBuf; pub(crate) const CONFIG_DIR_NAME: &str = ".toolpath"; pub(crate) const CONFIG_DIR_ENV: &str = "TOOLPATH_CONFIG_DIR"; +/// The artifact manifest under the config dir (see `sync::engine`). +pub(crate) const MANIFEST_FILE_NAME: &str = "manifest.json"; +/// Sibling advisory lock serializing manifest writers. A separate +/// file because the manifest itself is replaced by rename on every +/// write, which would drop any lock held on it. +pub(crate) const MANIFEST_LOCK_FILE_NAME: &str = "manifest.json.lock"; + /// The configured toolpath config directory (default `~/.toolpath`, /// overridable via `$TOOLPATH_CONFIG_DIR`). pub(crate) fn config_dir() -> Result { diff --git a/crates/path-cli/src/derive.rs b/crates/path-cli/src/derive.rs index 4ce230e3..d9b09e89 100644 --- a/crates/path-cli/src/derive.rs +++ b/crates/path-cli/src/derive.rs @@ -8,12 +8,23 @@ use anyhow::Result; use std::path::PathBuf; use toolpath::v1::Graph; -use crate::artifact::ArtifactType; +use crate::artifact::{ArtifactRef, ArtifactType, claude_chain_stamp, stat_stamp}; use crate::cache::make_id; pub(crate) struct DerivedDoc { pub(crate) cache_id: String, pub(crate) doc: Graph, + /// Identity + source stamp for the sync manifest, captured *before* + /// the source was read (so a write racing the derive re-syncs next + /// run). `None` for sources the manifest doesn't track (github, + /// pathbase) and for bulk `--all` derives that no longer know + /// per-artifact sources. + pub(crate) provenance: Option, +} + +/// Extract the inner identifier from a graph (without source prefix). +pub(crate) fn doc_inner_id(doc: &Graph) -> String { + doc.graph.id.clone() } /// Derive a single Claude conversation given an explicit project + session. @@ -31,6 +42,16 @@ pub(crate) fn derive_claude_session_with( project: &str, session: &str, ) -> Result { + // Fingerprint the whole session chain, and key the manifest record + // by the chain head (the rotation-stable id sync enumerates), so a + // caller passing a successor-segment id still records the artifact + // sync will look for. + let (modified, size) = claude_chain_stamp(manager, project, session); + let artifact_id = manager + .session_chain(project, session) + .ok() + .and_then(|chain| chain.into_iter().next()) + .unwrap_or_else(|| session.to_string()); // The caller's project string often comes from claude's lossy dir // slugs ('/', '_', '.' all collapsed); leaving it out of the derive // lets path.base come from the session's own recorded cwd instead. @@ -57,6 +78,13 @@ pub(crate) fn derive_claude_session_with( Ok(DerivedDoc { cache_id, doc: Graph::from_path(path), + provenance: Some(ArtifactRef { + artifact_type: ArtifactType::Claude, + id: artifact_id, + path: Some(project.to_string()), + modified, + size, + }), }) } @@ -71,6 +99,22 @@ pub(crate) fn derive_gemini_session_with( project: &str, session: &str, ) -> Result { + let entry = manager + .resolver() + .list_session_entries(project) + .ok() + .and_then(|entries| { + entries + .into_iter() + .find(|e| e.id == session || e.session_uuid.as_deref() == Some(session)) + }); + let (artifact_id, (modified, size)) = match &entry { + Some(e) => ( + e.session_uuid.clone().unwrap_or_else(|| e.id.clone()), + stat_stamp(&e.path), + ), + None => (session.to_string(), (None, None)), + }; // Maximal ingest: thinking is always derived into the cache. let cfg = toolpath_gemini::derive::DeriveConfig { project_path: Some(project.to_string()), @@ -84,6 +128,13 @@ pub(crate) fn derive_gemini_session_with( Ok(DerivedDoc { cache_id, doc: Graph::from_path(path), + provenance: Some(ArtifactRef { + artifact_type: ArtifactType::Gemini, + id: artifact_id, + path: Some(project.to_string()), + modified, + size, + }), }) } @@ -97,6 +148,14 @@ pub(crate) fn derive_codex_session_with( manager: &toolpath_codex::CodexConvo, session: &str, ) -> Result { + let file = manager.resolver().find_rollout_file(session).ok(); + let (modified, size) = file.as_deref().map(stat_stamp).unwrap_or((None, None)); + let artifact_id = file + .as_deref() + .and_then(|f| f.file_stem()) + .and_then(|stem| stem.to_str()) + .map(|stem| toolpath_codex::session_id_from_stem(stem).to_string()) + .unwrap_or_else(|| session.to_string()); let config = toolpath_codex::derive::DeriveConfig { project_path: None }; let s = manager .read_session(session) @@ -106,6 +165,13 @@ pub(crate) fn derive_codex_session_with( Ok(DerivedDoc { cache_id, doc: Graph::from_path(path), + provenance: Some(ArtifactRef { + artifact_type: ArtifactType::Codex, + id: artifact_id, + path: None, + modified, + size, + }), }) } @@ -119,6 +185,11 @@ pub(crate) fn derive_copilot_session_with( manager: &toolpath_copilot::CopilotConvo, session: &str, ) -> Result { + let (modified, size) = manager + .resolver() + .events_file(session) + .map(|p| stat_stamp(&p)) + .unwrap_or((None, None)); let config = toolpath_copilot::derive::DeriveConfig { project_path: None }; let s = manager .read_session(session) @@ -128,6 +199,13 @@ pub(crate) fn derive_copilot_session_with( Ok(DerivedDoc { cache_id, doc: Graph::from_path(path), + provenance: Some(ArtifactRef { + artifact_type: ArtifactType::Copilot, + id: session.to_string(), + path: None, + modified, + size, + }), }) } @@ -151,6 +229,12 @@ pub(crate) fn derive_opencode_session_with( session: &str, no_snapshot_diffs: bool, ) -> Result { + let modified = manager + .io() + .list_sessions(None) + .ok() + .and_then(|sessions| sessions.into_iter().find(|s| s.id == session)) + .and_then(|s| s.last_activity()); let config = toolpath_opencode::derive::DeriveConfig { no_snapshot_diffs, ..Default::default() @@ -164,6 +248,13 @@ pub(crate) fn derive_opencode_session_with( Ok(DerivedDoc { cache_id, doc: Graph::from_path(path), + provenance: Some(ArtifactRef { + artifact_type: ArtifactType::Opencode, + id: session.to_string(), + path: None, + modified, + size: None, + }), }) } @@ -179,6 +270,16 @@ pub(crate) fn derive_cursor_session_with( manager: &toolpath_cursor::CursorConvo, session: &str, ) -> Result { + let modified = manager + .io() + .read_composer_headers() + .ok() + .and_then(|h| { + h.all_composers + .into_iter() + .find(|c| c.composer_id == session) + }) + .and_then(|c| c.last_updated_at_utc()); let s = manager .read_session(session) .map_err(|e| anyhow::anyhow!("{}", e))?; @@ -188,6 +289,13 @@ pub(crate) fn derive_cursor_session_with( Ok(DerivedDoc { cache_id, doc: Graph::from_path(path), + provenance: Some(ArtifactRef { + artifact_type: ArtifactType::Cursor, + id: session.to_string(), + path: None, + modified, + size: None, + }), }) } @@ -212,13 +320,36 @@ pub(crate) fn derive_pi_session_with( project: &str, session: &str, ) -> Result { + let file = toolpath_pi::reader::list_session_files(manager.resolver(), project) + .ok() + .and_then(|files| { + files.into_iter().find(|f| { + toolpath_pi::reader::peek_header(f).is_ok_and(|h| h.id == session) + || f.file_stem() + .and_then(|stem| stem.to_str()) + .and_then(|stem| stem.split_once('_')) + .is_some_and(|(_, rest)| rest == session) + }) + }); + let (modified, size) = file.as_deref().map(stat_stamp).unwrap_or((None, None)); + let artifact_id = session.to_string(); let config = toolpath_pi::DeriveConfig::default(); let session = manager .read_session(project, session) .map_err(|e| anyhow::anyhow!("{}", e))?; let doc = Graph::from_path(toolpath_pi::derive::derive_path(&session, &config)); let cache_id = make_id(ArtifactType::Pi.name(), &doc_inner_id(&doc)); - Ok(DerivedDoc { cache_id, doc }) + Ok(DerivedDoc { + cache_id, + doc, + provenance: Some(ArtifactRef { + artifact_type: ArtifactType::Pi, + id: artifact_id, + path: Some(project.to_string()), + modified, + size, + }), + }) } /// Fetch a Pathbase ref (`https://host/u/owner/repos/repo/graphs/` @@ -241,7 +372,11 @@ pub(crate) fn pathbase_fetch_to_doc(target: &str, url_flag: Option<&str>) -> Res let cache_id = crate::cache::pathbase_cache_id(&owner, &repo, &id); let doc = Graph::from_json(&body) .map_err(|e| anyhow::anyhow!("server returned a non-toolpath document: {e}"))?; - Ok(DerivedDoc { cache_id, doc }) + Ok(DerivedDoc { + cache_id, + doc, + provenance: None, + }) } /// What the user pointed at on the import side. Pathbase 1.1+ @@ -342,11 +477,6 @@ fn extract_triple(segs: &[&str]) -> Option { }) } -/// Extract the inner identifier from a graph (without source prefix). -pub(crate) fn doc_inner_id(doc: &Graph) -> String { - doc.graph.id.clone() -} - #[cfg(all(test, not(target_os = "emscripten")))] mod tests { use super::*; diff --git a/crates/path-cli/src/lib.rs b/crates/path-cli/src/lib.rs index cd364afa..14ed9bba 100644 --- a/crates/path-cli/src/lib.rs +++ b/crates/path-cli/src/lib.rs @@ -38,6 +38,7 @@ mod query; mod schema; #[cfg(all(not(target_os = "emscripten"), feature = "embedded-picker"))] mod skim_picker; +mod sync; mod term; use anyhow::Result; diff --git a/crates/path-cli/src/sync.rs b/crates/path-cli/src/sync.rs new file mode 100644 index 00000000..0067ed66 --- /dev/null +++ b/crates/path-cli/src/sync.rs @@ -0,0 +1,20 @@ +//! `path p cache sync` — incremental ingestion of artifacts into the +//! document cache. +//! +//! Sync enumerates artifacts across the requested types (today all six +//! are agent-session providers), compares each against the sync +//! manifest at `$CONFIG_DIR/manifest.json`, and derives + caches only what +//! is new or changed. Change detection is stat-level: the fingerprint +//! is the source file's mtime + size (or the database row's updated-at +//! for the SQLite-backed providers), so deciding "nothing changed" +//! never reads session bodies. Artifacts deleted upstream keep both +//! their cache document and their manifest record. + +#[cfg(not(target_os = "emscripten"))] +pub(crate) use engine::*; + +#[cfg(not(target_os = "emscripten"))] +mod engine; + +#[cfg(not(target_os = "emscripten"))] +pub(crate) mod sources; diff --git a/crates/path-cli/src/sync/engine.rs b/crates/path-cli/src/sync/engine.rs new file mode 100644 index 00000000..10dec31b --- /dev/null +++ b/crates/path-cli/src/sync/engine.rs @@ -0,0 +1,1057 @@ +//! The sync engine: the manifest at `$CONFIG_DIR/manifest.json`, the +//! stat-gated ingestion loop, and the record surfaces `p import`, +//! `share`, and `p cache rm` use to keep the manifest honest. + +use anyhow::{Context, Result, anyhow}; +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; +use std::collections::BTreeMap; +use std::path::PathBuf; + +use super::sources::{self, ArtifactSource}; +use crate::artifact::{ArtifactRef, ArtifactType}; +use crate::cache::write_cached; +use crate::config::{MANIFEST_FILE_NAME, MANIFEST_LOCK_FILE_NAME, config_dir}; +use crate::harness::HarnessBundle; + +/// How many manifest writes accumulate before a mid-run checkpoint. +/// Small enough that an interrupted first sync loses at most a few +/// records (the cache docs themselves survive either way); large +/// enough that manifest serialization stays noise against the +/// derives it punctuates. +const MANIFEST_CHECKPOINT_EVERY_WRITES: usize = 10; + +/// What the manifest remembers about one known artifact. A record +/// with a `cache_id` is materialized in the cache; one without is +/// merely known — evicted by `p cache rm`. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub(crate) struct SyncRecord { + /// Filesystem path the artifact is keyed under: the project + /// directory for path-keyed providers, the recorded cwd / + /// workspace for the others (when known). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(crate) path: Option, + /// Cache entry the derived document was written to; `None` + /// when the artifact is known but not cached. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(crate) cache_id: Option, + /// Fingerprint: source mtime (file providers) or updated-at + /// (DB providers) at sync time. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(crate) modified: Option>, + /// Fingerprint: source file size at sync time. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub(crate) size: Option, + pub(crate) synced_at: DateTime, +} + +/// The sync manifest: artifact type (`"claude"`, `"codex"`, …) → +/// artifact id → record. Kept as `BTreeMap`s so the JSON on disk is +/// stably ordered. +pub(crate) type Manifest = BTreeMap>; + +/// Per-type tally of what one sync run did. +#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)] +pub(crate) struct SyncOutcome { + pub(crate) new: usize, + pub(crate) updated: usize, + pub(crate) unchanged: usize, + pub(crate) failed: usize, +} + +impl SyncOutcome { + fn total(&self) -> usize { + self.new + self.updated + self.unchanged + self.failed + } +} + +pub(crate) fn run(types: Vec) -> Result<()> { + let explicit = !types.is_empty(); + let types = resolve_types(&types); + let bundle = HarnessBundle::from_environment(); + let outcomes = sync_bundle(&bundle, &types)?; + eprint!("{}", render_summary(&outcomes, explicit)); + Ok(()) +} + +/// Explicit args → dedup'd type list; no args → every type. +fn resolve_types(args: &[ArtifactType]) -> Vec { + if args.is_empty() { + return ArtifactType::ALL.to_vec(); + } + let mut out: Vec = Vec::with_capacity(args.len()); + for &t in args { + if !out.contains(&t) { + out.push(t); + } + } + out +} + +/// Sync the given artifact types from `bundle` into the cache, +/// newest artifacts first. The manifest is checkpointed every few +/// writes (see [`MANIFEST_CHECKPOINT_EVERY_WRITES`]), so an interrupted run keeps +/// nearly everything it derived. Reads come from a point-in-time +/// snapshot; each checkpoint merges only the records this run +/// wrote, under the manifest lock, so concurrent invocations +/// (query auto-syncs, imports) union their records instead of +/// clobbering each other. +pub(crate) fn sync_bundle( + bundle: &HarnessBundle, + types: &[ArtifactType], +) -> Result> { + let manifest = load_manifest()?; + let mut out = Vec::with_capacity(types.len()); + for &artifact_type in types { + // Types with no source in this bundle — an uninstalled + // provider, or git, which is recorded but never discovered — + // sync as a no-op. + let Some(source) = sources::source_for(bundle, artifact_type) else { + out.push((artifact_type, SyncOutcome::default())); + continue; + }; + let artifacts = source.enumerate(); + let records = manifest + .get(artifact_type.name()) + .cloned() + .unwrap_or_default(); + let outcome = sync_artifacts(source.as_ref(), &artifacts, &records)?; + out.push((artifact_type, outcome)); + } + Ok(out) +} + +/// The stat gate: a materialized record whose real stamps match the +/// artifact needs nothing — no read, no scope check. All-`None` stamps +/// mean freshness is unknowable; only a real stamp can vouch +/// (mirrors `record_is_current`). +fn is_unchanged(rec: Option<&SyncRecord>, artifact: &ArtifactRef) -> bool { + rec.is_some_and(|rec| { + (rec.modified.is_some() || rec.size.is_some()) + && rec.modified == artifact.modified + && rec.size == artifact.size + && rec + .cache_id + .as_deref() + .is_some_and(|id| crate::cache::cache_path(id).is_ok_and(|p| p.exists())) + }) +} + +/// Artifacts newest-first (unstamped last), so an interrupted run has +/// spent its time on the sessions the user most likely wants. +fn newest_first(artifacts: &[ArtifactRef]) -> Vec<&ArtifactRef> { + let mut order: Vec<&ArtifactRef> = artifacts.iter().collect(); + order.sort_by(|a, b| b.modified.cmp(&a.modified)); + order +} + +/// Merge staged records into the manifest under the lock and clear +/// the stage. +fn flush_writes(pending: &mut BTreeMap<&'static str, BTreeMap>) -> Result<()> { + if pending.is_empty() { + return Ok(()); + } + let batch = std::mem::take(pending); + update_manifest(move |manifest| { + for (name, records) in batch { + manifest + .entry(name.to_string()) + .or_default() + .extend(records); + } + }) +} + +/// Sync one source's artifacts against a snapshot of its manifest +/// records, newest first. Records are checkpointed to the manifest every +/// [`MANIFEST_CHECKPOINT_EVERY_WRITES`] writes (and once more at the end), so an +/// interrupted run keeps nearly everything it derived. Derivation +/// failures are warned and tallied, not fatal; cache-write failures +/// (disk, permissions) abort. +fn sync_artifacts( + source: &dyn ArtifactSource, + artifacts: &[ArtifactRef], + records: &BTreeMap, +) -> Result { + let mut outcome = SyncOutcome::default(); + // Evaluate the stat gate once per artifact: the pass feeds both the + // progress denominator and the loop's skip decision. + let order: Vec<(&ArtifactRef, bool)> = newest_first(artifacts) + .into_iter() + .map(|artifact| (artifact, is_unchanged(records.get(&artifact.id), artifact))) + .collect(); + let pending_total = order.iter().filter(|(_, unchanged)| !unchanged).count(); + let mut progress = Progress::start( + artifacts + .first() + .map(|a| a.artifact_type.padded_name()) + .unwrap_or_default(), + pending_total, + ); + let mut writes: BTreeMap<&'static str, BTreeMap> = BTreeMap::new(); + let mut unflushed = 0usize; + for (artifact, unchanged) in order { + if unchanged { + outcome.unchanged += 1; + continue; + } + let existing = records.get(&artifact.id); + let is_new = existing.is_none(); + let stage = |writes: &mut BTreeMap<&'static str, BTreeMap>, + record: SyncRecord| { + writes + .entry(artifact.artifact_type.name()) + .or_default() + .insert(artifact.id.clone(), record); + }; + // An artifact without a path must not erase one an earlier + // run recorded. + let memoized_path = artifact + .path + .clone() + .or_else(|| existing.and_then(|r| r.path.clone())); + match source.derive(artifact) { + Ok(derived) => { + // force: sync owns refresh semantics — a re-sync or a + // prior manual `p import` of the same session must not + // error on the existing cache entry. + write_cached(&derived.cache_id, &derived.doc, true)?; + stage( + &mut writes, + SyncRecord { + path: memoized_path, + cache_id: Some(derived.cache_id), + // The stamp was taken before the derive read the + // source, so a write racing the derive re-syncs + // next run instead of going unnoticed. + modified: artifact.modified, + size: artifact.size, + synced_at: Utc::now(), + }, + ); + unflushed += 1; + if is_new { + outcome.new += 1; + } else { + outcome.updated += 1; + } + } + Err(e) => { + progress.interrupt(); + eprintln!( + "warning: sync {}: {}: {e}", + artifact.artifact_type.name(), + artifact.id + ); + outcome.failed += 1; + } + } + progress.tick(); + if unflushed >= MANIFEST_CHECKPOINT_EVERY_WRITES { + flush_writes(&mut writes)?; + unflushed = 0; + } + } + flush_writes(&mut writes)?; + progress.interrupt(); + Ok(outcome) +} + +/// Live sync progress on stderr: a `\r`-updating ` done/total` +/// line on a terminal, a plain line every 25 items otherwise. Only +/// artifacts needing work count toward the total — a no-op sync +/// draws nothing. +struct Progress { + label: String, + total: usize, + done: usize, + tty: bool, +} + +impl Progress { + fn start(label: String, total: usize) -> Self { + use std::io::IsTerminal; + let progress = Self { + label, + total, + done: 0, + tty: std::io::stderr().is_terminal(), + }; + progress.draw(); + progress + } + + fn line(&self) -> String { + format!("{} {}/{}", self.label, self.done, self.total) + } + + fn draw(&self) { + if self.total > 0 && self.tty { + eprint!("\r{}", self.line()); + } + } + + fn tick(&mut self) { + if self.total == 0 { + return; + } + self.done += 1; + if self.tty { + self.draw(); + } else if self.done.is_multiple_of(25) { + eprintln!("{}", self.line()); + } + } + + /// Clear the live line so a warning or summary prints clean; + /// the next `tick` redraws in full. + fn interrupt(&self) { + if self.total > 0 && self.tty { + eprint!("\r\x1b[2K"); + } + } +} + +/// One stderr line per artifact type. Types the user didn't name +/// are shown only when they had artifacts, so a default run doesn't +/// list every uninstalled provider. +fn render_summary(outcomes: &[(ArtifactType, SyncOutcome)], explicit: bool) -> String { + let mut s = String::new(); + for (artifact_type, o) in outcomes { + if o.total() == 0 && !explicit { + continue; + } + s.push_str(&format!( + "{} {} new, {} updated, {} unchanged", + artifact_type.padded_name(), + o.new, + o.updated, + o.unchanged + )); + if o.failed > 0 { + s.push_str(&format!(", {} failed", o.failed)); + } + s.push('\n'); + } + if s.is_empty() { + s.push_str("nothing to sync\n"); + } + s +} + +/// Record an externally-derived cache write (`p import`, `share`) in +/// the manifest, so sync doesn't re-derive what was just written. +pub(crate) fn record_artifact(artifact: &ArtifactRef, cache_id: &str) -> Result<()> { + update_manifest(|manifest| { + manifest + .entry(artifact.artifact_type.name().to_string()) + .or_default() + .insert( + artifact.id.clone(), + SyncRecord { + path: artifact.path.clone(), + cache_id: Some(cache_id.to_string()), + modified: artifact.modified, + size: artifact.size, + synced_at: Utc::now(), + }, + ); + }) +} + +/// Whether the manifest already records exactly this artifact state +/// under exactly this cache entry, with the doc present — i.e. a +/// write would reproduce what's already there. +pub(crate) fn record_is_current(artifact: &ArtifactRef, cache_id: &str) -> bool { + let Ok(manifest) = load_manifest() else { + return false; + }; + manifest + .get(artifact.artifact_type.name()) + .and_then(|records| records.get(&artifact.id)) + .is_some_and(|rec| { + rec.cache_id.as_deref() == Some(cache_id) + // None stamps mean freshness is unknowable (git); only + // a real, matching stamp can vouch for the cache entry. + && (rec.modified.is_some() || rec.size.is_some()) + && rec.modified == artifact.modified + && rec.size == artifact.size + && crate::cache::cache_path(cache_id).is_ok_and(|p| p.exists()) + }) +} + +/// The cache entry for an artifact, when the manifest says it is +/// materialized and a fresh stat shows its source unchanged since — +/// i.e. re-deriving would reproduce the cached doc byte-for-byte. +/// Used by `share` to upload straight from the cache. The stat +/// targets one artifact directly — no enumeration of its siblings. +pub(crate) fn fresh_cache_id( + bundle: &HarnessBundle, + artifact_type: ArtifactType, + project: Option<&str>, + id: &str, +) -> Option { + let manifest = load_manifest().ok()?; + let rec = manifest.get(artifact_type.name())?.get(id)?; + let cache_id = rec.cache_id.clone()?; + let (modified, size) = sources::source_for(bundle, artifact_type)?.stamp(project, id)?; + // None stamps mean freshness is unknowable; only a real, + // matching stamp can vouch for the cache entry. + ((rec.modified.is_some() || rec.size.is_some()) + && rec.modified == modified + && rec.size == size + && crate::cache::cache_path(&cache_id).is_ok_and(|p| p.exists())) + .then_some(cache_id) +} + +/// `p cache rm` eviction: the doc is gone, so any record pointing +/// at it downgrades to known-but-uncached (the artifact itself is +/// still real; the next in-scope sync re-materializes it). +pub(crate) fn evict_cache_id(cache_id: &str) -> Result<()> { + update_manifest(|manifest| { + for records in manifest.values_mut() { + for rec in records.values_mut() { + if rec.cache_id.as_deref() == Some(cache_id) { + rec.cache_id = None; + } + } + } + }) +} + +// ── manifest IO ──────────────────────────────────────────────────── + +fn manifest_path() -> Result { + Ok(config_dir()?.join(MANIFEST_FILE_NAME)) +} + +/// Take the exclusive advisory lock serializing manifest writers +/// across processes (query auto-syncs and imports can run +/// concurrently). A sibling lock file — never renamed, unlike the +/// manifest itself — held until the returned handle drops. +fn lock_manifest() -> Result { + let path = manifest_path()?; + let dir = path.parent().expect("manifest path has a parent"); + std::fs::create_dir_all(dir).with_context(|| format!("create {}", dir.display()))?; + let lock_path = dir.join(MANIFEST_LOCK_FILE_NAME); + let file = std::fs::File::create(&lock_path) + .with_context(|| format!("create {}", lock_path.display()))?; + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + let _ = std::fs::set_permissions(&lock_path, std::fs::Permissions::from_mode(0o600)); + } + file.lock() + .with_context(|| format!("lock {}", lock_path.display()))?; + Ok(file) +} + +/// One locked read-modify-write cycle against the manifest. Every +/// writer goes through here, so concurrent invocations merge their +/// records instead of clobbering each other's. +fn update_manifest(mutate: impl FnOnce(&mut Manifest)) -> Result<()> { + let _lock = lock_manifest()?; + let mut manifest = load_manifest()?; + mutate(&mut manifest); + save_manifest(&manifest) +} + +pub(crate) fn load_manifest() -> Result { + let path = manifest_path()?; + let json = match std::fs::read_to_string(&path) { + Ok(s) => s, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(Manifest::default()), + Err(e) => return Err(anyhow!("read {}: {e}", path.display())), + }; + serde_json::from_str(&json).with_context(|| { + format!( + "parse {}; delete it to re-sync from scratch", + path.display() + ) + }) +} + +/// Write the manifest atomically (temp file + rename) with the same +/// permissions as the rest of `$CONFIG_DIR`. +fn save_manifest(manifest: &Manifest) -> Result<()> { + let path = manifest_path()?; + let dir = path.parent().expect("manifest path has a parent"); + std::fs::create_dir_all(dir).with_context(|| format!("create {}", dir.display()))?; + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + let _ = std::fs::set_permissions(dir, std::fs::Permissions::from_mode(0o700)); + } + let json = serde_json::to_string_pretty(manifest)?; + let tmp = dir.join(format!("{MANIFEST_FILE_NAME}.{}.tmp", std::process::id())); + std::fs::write(&tmp, json).with_context(|| format!("write {}", tmp.display()))?; + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + std::fs::set_permissions(&tmp, std::fs::Permissions::from_mode(0o600)) + .with_context(|| format!("chmod 0600 {}", tmp.display()))?; + } + std::fs::rename(&tmp, &path) + .with_context(|| format!("rename {} → {}", tmp.display(), path.display())) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::config::{CONFIG_DIR_ENV, TEST_ENV_LOCK}; + use std::path::Path; + + /// Run `f` with `$TOOLPATH_CONFIG_DIR` pinned to `/.toolpath`; + /// `f` receives the tempdir root for building provider fixtures. + fn with_cfg R, R>(f: F) -> R { + let _g = TEST_ENV_LOCK.lock().unwrap_or_else(|e| e.into_inner()); + let temp = tempfile::tempdir().unwrap(); + let prev = std::env::var_os(CONFIG_DIR_ENV); + unsafe { + std::env::set_var(CONFIG_DIR_ENV, temp.path().join(".toolpath")); + } + let result = f(temp.path()); + unsafe { + match prev { + Some(v) => std::env::set_var(CONFIG_DIR_ENV, v), + None => std::env::remove_var(CONFIG_DIR_ENV), + } + } + result + } + + fn write_claude_session(home: &Path, project_slug: &str, session: &str, prompt: &str) { + let project_dir = home.join(".claude/projects").join(project_slug); + std::fs::create_dir_all(&project_dir).unwrap(); + let user = format!( + r#"{{"type":"user","uuid":"u-{session}","timestamp":"2024-01-02T00:00:00Z","cwd":"/test/project","message":{{"role":"user","content":"{prompt}"}}}}"# + ); + let asst = format!( + r#"{{"type":"assistant","uuid":"a-{session}","timestamp":"2024-01-02T00:00:01Z","message":{{"role":"assistant","content":"hi"}}}}"# + ); + std::fs::write( + project_dir.join(format!("{session}.jsonl")), + format!("{user}\n{asst}\n"), + ) + .unwrap(); + } + + fn claude_bundle(home: &Path) -> HarnessBundle { + let resolver = toolpath_claude::PathResolver::new().with_claude_dir(home.join(".claude")); + HarnessBundle { + claude: Some(toolpath_claude::ClaudeConvo::with_resolver(resolver)), + ..Default::default() + } + } + + fn cached_step_count(cache_id: &str) -> usize { + let path = crate::cache::cache_path(cache_id).unwrap(); + let json = std::fs::read_to_string(path).unwrap(); + let doc = toolpath::v1::Graph::from_json(&json).unwrap(); + doc.single_path().map(|p| p.steps.len()).unwrap_or(0) + } + + fn make_ref(artifact_type: ArtifactType, id: &str) -> ArtifactRef { + ArtifactRef { + artifact_type, + id: id.to_string(), + path: Some("/test/project".to_string()), + modified: None, + size: None, + } + } + + #[test] + fn manifest_roundtrips_and_missing_is_empty() { + with_cfg(|_| { + assert!(load_manifest().unwrap().is_empty()); + + let mut manifest = Manifest::default(); + manifest.entry("claude".to_string()).or_default().insert( + "sess-1".to_string(), + SyncRecord { + path: Some("/test/project".to_string()), + cache_id: Some("claude-p1".to_string()), + modified: Some("2024-01-02T00:00:01.123456789Z".parse().unwrap()), + size: Some(4096), + synced_at: "2026-07-09T00:00:00Z".parse().unwrap(), + }, + ); + save_manifest(&manifest).unwrap(); + assert_eq!(load_manifest().unwrap(), manifest); + }); + } + + #[cfg(unix)] + #[test] + fn manifest_file_is_0600() { + use std::os::unix::fs::PermissionsExt; + with_cfg(|_| { + save_manifest(&Manifest::default()).unwrap(); + let mode = std::fs::metadata(manifest_path().unwrap()) + .unwrap() + .permissions() + .mode() + & 0o777; + assert_eq!(mode, 0o600); + }); + } + + #[test] + fn corrupt_manifest_errors_with_hint() { + with_cfg(|_| { + save_manifest(&Manifest::default()).unwrap(); + std::fs::write(manifest_path().unwrap(), "not json").unwrap(); + let err = load_manifest().unwrap_err(); + assert!(err.to_string().contains("re-sync from scratch")); + }); + } + + #[test] + fn enumerated_claude_sessions_are_stamped() { + with_cfg(|home| { + write_claude_session(home, "-test-project", "sess-aaa", "Add a feature"); + let bundle = claude_bundle(home); + let source = sources::source_for(&bundle, ArtifactType::Claude).unwrap(); + let artifacts = source.enumerate(); + assert_eq!(artifacts.len(), 1); + assert_eq!(artifacts[0].id, "sess-aaa"); + assert_eq!(artifacts[0].path.as_deref(), Some("/test/project")); + assert!( + artifacts[0].modified.is_some(), + "file mtime must be stamped" + ); + assert!(artifacts[0].size.unwrap() > 0, "file size must be stamped"); + }); + } + + #[test] + fn first_sync_ingests_then_second_is_unchanged() { + with_cfg(|home| { + write_claude_session(home, "-test-project", "sess-aaa", "Add a feature"); + write_claude_session(home, "-test-project", "sess-bbb", "Fix a bug"); + let bundle = claude_bundle(home); + + let outcomes = sync_bundle(&bundle, &[ArtifactType::Claude]).unwrap(); + assert_eq!(outcomes.len(), 1); + let (_, first) = outcomes[0]; + assert_eq!( + (first.new, first.updated, first.unchanged, first.failed), + (2, 0, 0, 0) + ); + + let manifest = load_manifest().unwrap(); + let records = manifest.get("claude").unwrap(); + assert_eq!(records.len(), 2); + let rec = records.get("sess-aaa").unwrap(); + assert_eq!(rec.path.as_deref(), Some("/test/project")); + assert!(rec.modified.is_some()); + assert!(rec.size.is_some()); + let cache_id = rec + .cache_id + .as_deref() + .expect("synced record is materialized"); + assert!( + crate::cache::cache_path(cache_id).unwrap().exists(), + "cache doc must exist for {cache_id}" + ); + + let (_, second) = sync_bundle(&bundle, &[ArtifactType::Claude]).unwrap()[0]; + assert_eq!( + (second.new, second.updated, second.unchanged, second.failed), + (0, 0, 2, 0) + ); + }); + } + + #[test] + fn changed_session_is_rederived() { + with_cfg(|home| { + write_claude_session(home, "-test-project", "sess-aaa", "Add a feature"); + let bundle = claude_bundle(home); + sync_bundle(&bundle, &[ArtifactType::Claude]).unwrap(); + + let cache_id = load_manifest().unwrap()["claude"]["sess-aaa"] + .cache_id + .clone() + .expect("synced record is materialized"); + let steps_before = cached_step_count(&cache_id); + + // Session continues: a later user turn lands in the file, + // changing its size (and mtime). + let file = home.join(".claude/projects/-test-project/sess-aaa.jsonl"); + let mut body = std::fs::read_to_string(&file).unwrap(); + body.push_str( + r#"{"type":"user","uuid":"u-2","timestamp":"2024-01-02T00:05:00Z","cwd":"/test/project","message":{"role":"user","content":"And another thing"}}"#, + ); + body.push('\n'); + std::fs::write(&file, body).unwrap(); + + let (_, outcome) = sync_bundle(&bundle, &[ArtifactType::Claude]).unwrap()[0]; + assert_eq!( + ( + outcome.new, + outcome.updated, + outcome.unchanged, + outcome.failed + ), + (0, 1, 0, 0) + ); + assert!( + cached_step_count(&cache_id) > steps_before, + "re-derived doc must contain the appended turn" + ); + }); + } + + #[test] + fn sync_touches_only_requested_types() { + with_cfg(|home| { + write_claude_session(home, "-test-project", "sess-aaa", "Add a feature"); + let bundle = claude_bundle(home); + + let outcomes = sync_bundle(&bundle, &[ArtifactType::Codex]).unwrap(); + assert_eq!(outcomes[0].1, SyncOutcome::default()); + assert!( + load_manifest().unwrap().is_empty(), + "codex-only sync must not ingest claude sessions" + ); + }); + } + + #[test] + fn sync_overwrites_cache_entry_it_does_not_remember() { + with_cfg(|home| { + write_claude_session(home, "-test-project", "sess-aaa", "Add a feature"); + let bundle = claude_bundle(home); + sync_bundle(&bundle, &[ArtifactType::Claude]).unwrap(); + + // Losing the manifest (or a prior manual `p import`) leaves a + // cache entry sync doesn't know about; re-syncing must + // overwrite it, not die on the exists-check. + std::fs::remove_file(manifest_path().unwrap()).unwrap(); + let (_, outcome) = sync_bundle(&bundle, &[ArtifactType::Claude]).unwrap()[0]; + assert_eq!((outcome.new, outcome.failed), (1, 0)); + }); + } + + #[test] + fn failed_derivation_is_tallied_and_skipped() { + with_cfg(|home| { + write_claude_session(home, "-test-project", "sess-aaa", "Add a feature"); + let bundle = claude_bundle(home); + let source = sources::source_for(&bundle, ArtifactType::Claude).unwrap(); + let mut artifacts = source.enumerate(); + artifacts.push(make_ref(ArtifactType::Claude, "does-not-exist")); + + let outcome = sync_artifacts(source.as_ref(), &artifacts, &BTreeMap::new()).unwrap(); + assert_eq!((outcome.new, outcome.failed), (1, 1)); + let records = &load_manifest().unwrap()["claude"]; + assert!(records.contains_key("sess-aaa")); + assert!( + !records.contains_key("does-not-exist"), + "failed artifacts must not be recorded as synced" + ); + }); + } + + #[test] + fn rotated_session_resyncs_under_its_head_id() { + with_cfg(|home| { + write_claude_session(home, "-test-project", "sess-aaa", "Add a feature"); + let bundle = claude_bundle(home); + sync_bundle(&bundle, &[ArtifactType::Claude]).unwrap(); + let cache_id = load_manifest().unwrap()["claude"]["sess-aaa"] + .cache_id + .clone() + .unwrap(); + let steps_before = cached_step_count(&cache_id); + + // The session rotates: a successor file whose first entry + // carries the predecessor's sessionId (the bridge). + // Appends land here; sess-aaa.jsonl never changes again. + std::fs::write( + home.join(".claude/projects/-test-project/sess-bbb.jsonl"), + concat!( + r#"{"type":"user","uuid":"u-b0","timestamp":"2024-01-02T01:00:00Z","sessionId":"sess-aaa","cwd":"/test/project","message":{"role":"user","content":"bridge"}}"#, + "\n", + r#"{"type":"user","uuid":"u-b1","timestamp":"2024-01-02T01:00:01Z","sessionId":"sess-bbb","cwd":"/test/project","message":{"role":"user","content":"after rotation"}}"#, + "\n", + ), + ) + .unwrap(); + + let (_, outcome) = sync_bundle(&bundle, &[ArtifactType::Claude]).unwrap()[0]; + assert_eq!( + (outcome.new, outcome.updated, outcome.unchanged), + (0, 1, 0), + "the chain must re-sync under its head id, not read as unchanged" + ); + let manifest = load_manifest().unwrap(); + assert!( + !manifest["claude"].contains_key("sess-bbb"), + "successor segments are not separate artifacts" + ); + assert!( + cached_step_count(&cache_id) > steps_before, + "post-rotation turns must reach the cached doc" + ); + + // And the grown chain settles: a third sync is a no-op. + let (_, again) = sync_bundle(&bundle, &[ArtifactType::Claude]).unwrap()[0]; + assert_eq!((again.updated, again.unchanged), (0, 1)); + }); + } + + #[test] + fn all_none_stamps_never_read_as_unchanged() { + with_cfg(|home| { + write_claude_session(home, "-test-project", "sess-aaa", "Add a feature"); + let bundle = claude_bundle(home); + sync_bundle(&bundle, &[ArtifactType::Claude]).unwrap(); + + // A record whose stamps are all None (stat failed when it + // was written) must not match a stub whose stat also + // failed — unknowable freshness re-derives. + let mut records = load_manifest().unwrap()["claude"].clone(); + let rec = records.get_mut("sess-aaa").unwrap(); + rec.modified = None; + rec.size = None; + let artifact = make_ref(ArtifactType::Claude, "sess-aaa"); + let source = sources::source_for(&bundle, ArtifactType::Claude).unwrap(); + let outcome = sync_artifacts(source.as_ref(), &[artifact], &records).unwrap(); + assert_eq!((outcome.updated, outcome.unchanged), (1, 0)); + }); + } + + #[test] + fn recorded_import_is_unchanged_to_the_next_sync() { + with_cfg(|home| { + write_claude_session(home, "-test-project", "sess-aaa", "Add a feature"); + let bundle = claude_bundle(home); + + // What `p import` does: derive with provenance, write the + // cache, record the stub. + let derived = crate::derive::derive_claude_session_with( + bundle.claude.as_ref().unwrap(), + "/test/project", + "sess-aaa", + ) + .unwrap(); + let artifact = derived.provenance.as_ref().unwrap(); + assert_eq!(artifact.id, "sess-aaa"); + assert!(artifact.modified.is_some() && artifact.size.is_some()); + crate::cache::write_cached(&derived.cache_id, &derived.doc, true).unwrap(); + record_artifact(artifact, &derived.cache_id).unwrap(); + + // The import's stamp must match sync's own enumeration. + let (_, outcome) = sync_bundle(&bundle, &[ArtifactType::Claude]).unwrap()[0]; + assert_eq!( + ( + outcome.new, + outcome.updated, + outcome.unchanged, + outcome.failed + ), + (0, 0, 1, 0) + ); + }); + } + + #[test] + fn evicted_cache_entry_rematerializes_on_next_sync() { + with_cfg(|home| { + write_claude_session(home, "-test-project", "sess-aaa", "Add a feature"); + let bundle = claude_bundle(home); + sync_bundle(&bundle, &[ArtifactType::Claude]).unwrap(); + let cache_id = load_manifest().unwrap()["claude"]["sess-aaa"] + .cache_id + .clone() + .unwrap(); + + // `p cache rm`: doc removed, record downgraded to known. + crate::cache::remove_cached(&cache_id).unwrap(); + evict_cache_id(&cache_id).unwrap(); + assert!( + load_manifest().unwrap()["claude"]["sess-aaa"] + .cache_id + .is_none() + ); + + let (_, outcome) = sync_bundle(&bundle, &[ArtifactType::Claude]).unwrap()[0]; + assert_eq!((outcome.new, outcome.updated), (0, 1)); + assert!( + crate::cache::cache_path(&cache_id).unwrap().exists(), + "evicted artifact re-materializes" + ); + }); + } + + #[test] + fn manually_deleted_doc_is_restored_even_with_stale_record() { + with_cfg(|home| { + write_claude_session(home, "-test-project", "sess-aaa", "Add a feature"); + let bundle = claude_bundle(home); + sync_bundle(&bundle, &[ArtifactType::Claude]).unwrap(); + let cache_id = load_manifest().unwrap()["claude"]["sess-aaa"] + .cache_id + .clone() + .unwrap(); + + // Doc deleted behind the CLI's back: the record still claims + // materialization, but sync verifies the doc exists. + let doc = crate::cache::cache_path(&cache_id).unwrap(); + std::fs::remove_file(&doc).unwrap(); + let (_, outcome) = sync_bundle(&bundle, &[ArtifactType::Claude]).unwrap()[0]; + assert_eq!((outcome.new, outcome.updated), (0, 1)); + assert!(doc.exists()); + }); + } + + #[test] + fn fresh_cache_id_tracks_source_and_eviction() { + with_cfg(|home| { + write_claude_session(home, "-test-project", "sess-aaa", "Add a feature"); + let bundle = claude_bundle(home); + + // Nothing synced yet: no fresh copy. + assert!( + fresh_cache_id( + &bundle, + ArtifactType::Claude, + Some("/test/project"), + "sess-aaa" + ) + .is_none() + ); + + sync_bundle(&bundle, &[ArtifactType::Claude]).unwrap(); + let cache_id = fresh_cache_id( + &bundle, + ArtifactType::Claude, + Some("/test/project"), + "sess-aaa", + ) + .expect("synced artifact is fresh"); + + // Source grows: stale until re-synced. + let file = home.join(".claude/projects/-test-project/sess-aaa.jsonl"); + let mut body = std::fs::read_to_string(&file).unwrap(); + body.push_str( + r#"{"type":"user","uuid":"u-2","timestamp":"2024-01-02T00:05:00Z","cwd":"/test/project","message":{"role":"user","content":"more"}}"#, + ); + body.push('\n'); + std::fs::write(&file, body).unwrap(); + assert!( + fresh_cache_id( + &bundle, + ArtifactType::Claude, + Some("/test/project"), + "sess-aaa" + ) + .is_none() + ); + sync_bundle(&bundle, &[ArtifactType::Claude]).unwrap(); + assert!( + fresh_cache_id( + &bundle, + ArtifactType::Claude, + Some("/test/project"), + "sess-aaa" + ) + .is_some() + ); + + // Evicted: known but not materialized, so not fresh. + crate::cache::remove_cached(&cache_id).unwrap(); + evict_cache_id(&cache_id).unwrap(); + assert!( + fresh_cache_id( + &bundle, + ArtifactType::Claude, + Some("/test/project"), + "sess-aaa" + ) + .is_none() + ); + }); + } + + #[test] + fn newest_first_orders_by_mtime_with_unstamped_last() { + let mut old = make_ref(ArtifactType::Claude, "old"); + old.modified = Some("2026-01-01T00:00:00Z".parse().unwrap()); + let mut new = make_ref(ArtifactType::Claude, "new"); + new.modified = Some("2026-07-01T00:00:00Z".parse().unwrap()); + let unstamped = make_ref(ArtifactType::Claude, "unstamped"); + + let stubs = vec![old, unstamped, new]; + let ids: Vec<&str> = newest_first(&stubs).iter().map(|s| s.id.as_str()).collect(); + assert_eq!(ids, vec!["new", "old", "unstamped"]); + } + + #[test] + fn progress_line_counts_only_pending_work() { + let mut progress = Progress { + label: crate::artifact::ArtifactType::Claude.padded_name(), + total: 3, + done: 0, + tty: false, + }; + progress.tick(); + progress.tick(); + assert_eq!(progress.line(), "claude 2/3"); + } + + #[test] + fn resolve_types_defaults_to_all_and_dedups() { + assert_eq!(resolve_types(&[]), ArtifactType::ALL.to_vec()); + assert_eq!( + resolve_types(&[ + ArtifactType::Codex, + ArtifactType::Claude, + ArtifactType::Codex + ]), + vec![ArtifactType::Codex, ArtifactType::Claude] + ); + } + + #[test] + fn render_summary_hides_empty_types_unless_explicit() { + let outcomes = vec![ + ( + ArtifactType::Claude, + SyncOutcome { + new: 2, + updated: 1, + unchanged: 3, + failed: 0, + }, + ), + (ArtifactType::Cursor, SyncOutcome::default()), + ]; + let default_run = render_summary(&outcomes, false); + assert!(default_run.contains("claude")); + assert!(!default_run.contains("cursor")); + + let explicit_run = render_summary(&outcomes, true); + assert!(explicit_run.contains("cursor")); + } + + #[test] + fn render_summary_shows_failures_and_empty_case() { + let outcomes = vec![( + ArtifactType::Codex, + SyncOutcome { + new: 0, + updated: 0, + unchanged: 1, + failed: 2, + }, + )]; + let s = render_summary(&outcomes, false); + assert!(s.contains("2 failed")); + + assert_eq!(render_summary(&[], false), "nothing to sync\n"); + } +} diff --git a/crates/path-cli/src/sync/sources.rs b/crates/path-cli/src/sync/sources.rs new file mode 100644 index 00000000..ba76608c --- /dev/null +++ b/crates/path-cli/src/sync/sources.rs @@ -0,0 +1,449 @@ +//! Per-provider artifact sources. [`ArtifactSource`] is the seam +//! between the sync engine and the providers: everything the engine +//! needs from one provider — enumerate its artifacts with stat-level +//! fingerprints, stat one artifact directly, derive one into a +//! document, peek at where one lives, compare directories in its key +//! space — sits behind the trait, one impl per provider. The engine +//! never matches on [`ArtifactType`]; provider details changing means +//! editing that provider's impl here, nothing else. + +use anyhow::{Result, anyhow}; +use chrono::{DateTime, Utc}; + +use crate::artifact::{ArtifactRef, ArtifactType, claude_chain_stamp, stat_stamp}; +use crate::derive::{self, DerivedDoc}; +use crate::harness::{ + HarnessBundle, is_not_found_claude, is_not_found_codex, is_not_found_cursor, + is_not_found_gemini, is_not_found_opencode, is_not_found_pi, +}; + +/// A source's stat-level fingerprint for one artifact: mtime (file +/// providers) or updated-at (DB providers), plus file size — each +/// `None` when unavailable. +pub(crate) type Stamp = (Option>, Option); + +/// One provider, as the sync engine sees it. +pub(crate) trait ArtifactSource { + /// Enumerate this provider's artifacts with stat-level + /// fingerprints. Never reads session bodies. Listing errors warn + /// and skip so one broken provider can't block a run. + fn enumerate(&self) -> Vec; + + /// Stat-level fingerprint for a single artifact, resolved + /// directly — the same stat targets as [`Self::enumerate`], so the + /// result compares against manifest records. `project` is required + /// by the path-keyed providers (claude/gemini/pi). + fn stamp(&self, project: Option<&str>, id: &str) -> Option; + + /// Derive one enumerated artifact into a cacheable document, + /// through the same manager it was enumerated from, so listing and + /// derivation always agree on provider roots. + fn derive(&self, artifact: &ArtifactRef) -> Result; +} + +/// The source for `t`'s artifacts, borrowing its manager from +/// `bundle`. `None` when the provider isn't in the bundle — and for +/// git, which is recorded by `p import` but never discovered: there is +/// no machine-wide registry of repos to enumerate. +pub(crate) fn source_for<'a>( + bundle: &'a HarnessBundle, + t: ArtifactType, +) -> Option> { + match t { + ArtifactType::Claude => Some(Box::new(ClaudeSource(bundle.claude.as_ref()?))), + ArtifactType::Gemini => Some(Box::new(GeminiSource(bundle.gemini.as_ref()?))), + ArtifactType::Codex => Some(Box::new(CodexSource(bundle.codex.as_ref()?))), + ArtifactType::Opencode => Some(Box::new(OpencodeSource(bundle.opencode.as_ref()?))), + ArtifactType::Cursor => Some(Box::new(CursorSource(bundle.cursor.as_ref()?))), + ArtifactType::Pi => Some(Box::new(PiSource(bundle.pi.as_ref()?))), + ArtifactType::Copilot => Some(Box::new(CopilotSource(bundle.copilot.as_ref()?))), + ArtifactType::Git => None, + } +} + +/// The project path a path-keyed artifact was enumerated under. +fn require_path(artifact: &ArtifactRef) -> Result<&str> { + artifact + .path + .as_deref() + .ok_or_else(|| anyhow!("artifact {} has no path", artifact.id)) +} + +// ── claude ───────────────────────────────────────────────────────── + +struct ClaudeSource<'a>(&'a toolpath_claude::ClaudeConvo); + +impl ArtifactSource for ClaudeSource<'_> { + /// Chain heads via `list_conversations` (bounded first-lines peek + /// per file, no full parse). The head is the chain's *oldest* + /// segment and its id is rotation-stable; the fingerprint covers + /// the whole chain (see `claude_chain_stamp`) because appends land + /// in the newest segment, not the head file. + fn enumerate(&self) -> Vec { + let mut out = Vec::new(); + let projects = match self.0.list_projects() { + Ok(ps) => ps, + Err(e) if is_not_found_claude(&e) => return out, + Err(e) => { + eprintln!("warning: claude enumeration failed: {e}"); + return out; + } + }; + for project in projects { + let heads = match self.0.list_conversations(&project) { + Ok(h) => h, + Err(e) => { + eprintln!("warning: claude project {project} failed: {e}"); + continue; + } + }; + for head in heads { + let (modified, size) = claude_chain_stamp(self.0, &project, &head); + out.push(ArtifactRef { + artifact_type: ArtifactType::Claude, + id: head, + path: Some(project.clone()), + modified, + size, + }); + } + } + out + } + + fn stamp(&self, project: Option<&str>, id: &str) -> Option { + Some(claude_chain_stamp(self.0, project?, id)) + } + + fn derive(&self, artifact: &ArtifactRef) -> Result { + derive::derive_claude_session_with(self.0, require_path(artifact)?, &artifact.id) + } +} + +// ── gemini ───────────────────────────────────────────────────────── + +struct GeminiSource<'a>(&'a toolpath_gemini::GeminiConvo); + +impl ArtifactSource for GeminiSource<'_> { + /// Session entries via a bounded identity peek (`toolpath-gemini` + /// reads at most the first 4 KiB of a main file); the fingerprint + /// stats the main file (or the orphan sub-agent directory). + fn enumerate(&self) -> Vec { + let mut out = Vec::new(); + let projects = match self.0.list_projects() { + Ok(ps) => ps, + Err(e) if is_not_found_gemini(&e) => return out, + Err(e) => { + eprintln!("warning: gemini enumeration failed: {e}"); + return out; + } + }; + for project in projects { + let entries = match self.0.resolver().list_session_entries(&project) { + Ok(entries) => entries, + Err(e) => { + eprintln!("warning: gemini project {project} failed: {e}"); + continue; + } + }; + for entry in entries { + let (modified, size) = stat_stamp(&entry.path); + out.push(ArtifactRef { + artifact_type: ArtifactType::Gemini, + id: entry.session_uuid.unwrap_or(entry.id), + path: Some(project.clone()), + modified, + size, + }); + } + } + out + } + + fn stamp(&self, project: Option<&str>, id: &str) -> Option { + let entries = self.0.resolver().list_session_entries(project?).ok()?; + let entry = entries + .into_iter() + .find(|e| e.id == id || e.session_uuid.as_deref() == Some(id))?; + Some(stat_stamp(&entry.path)) + } + + fn derive(&self, artifact: &ArtifactRef) -> Result { + derive::derive_gemini_session_with(self.0, require_path(artifact)?, &artifact.id) + } +} + +// ── codex ────────────────────────────────────────────────────────── + +struct CodexSource<'a>(&'a toolpath_codex::CodexConvo); + +impl ArtifactSource for CodexSource<'_> { + /// Rollout files, stat-only. The artifact id is the trailing UUID of + /// the filename stem (`rollout--`); `read_session` + /// accepts either the UUID or the full stem, so the fallback is safe. + fn enumerate(&self) -> Vec { + let mut out = Vec::new(); + let files = match self.0.io().list_rollout_files() { + Ok(f) => f, + Err(e) if is_not_found_codex(&e) => return out, + Err(e) => { + eprintln!("warning: codex enumeration failed: {e}"); + return out; + } + }; + for file in files { + let Some(stem) = file.file_stem().and_then(|s| s.to_str()) else { + continue; + }; + let id = toolpath_codex::session_id_from_stem(stem).to_string(); + let (modified, size) = stat_stamp(&file); + out.push(ArtifactRef { + artifact_type: ArtifactType::Codex, + id, + path: None, + modified, + size, + }); + } + out + } + + fn stamp(&self, _project: Option<&str>, id: &str) -> Option { + let file = self.0.resolver().find_rollout_file(id).ok()?; + Some(stat_stamp(&file)) + } + + fn derive(&self, artifact: &ArtifactRef) -> Result { + derive::derive_codex_session_with(self.0, &artifact.id) + } +} + +// ── opencode ─────────────────────────────────────────────────────── + +struct OpencodeSource<'a>(&'a toolpath_opencode::OpencodeConvo); + +impl ArtifactSource for OpencodeSource<'_> { + /// One header-only `SELECT` — `time_updated` is the fingerprint; no + /// message bodies are loaded. + fn enumerate(&self) -> Vec { + let mut out = Vec::new(); + let sessions = match self.0.io().list_sessions(None) { + Ok(s) => s, + Err(e) if is_not_found_opencode(&e) => return out, + Err(e) => { + eprintln!("warning: opencode enumeration failed: {e}"); + return out; + } + }; + for s in sessions { + out.push(ArtifactRef { + artifact_type: ArtifactType::Opencode, + modified: s.last_activity(), + path: Some(s.directory.to_string_lossy().into_owned()), + id: s.id, + size: None, + }); + } + out + } + + fn stamp(&self, _project: Option<&str>, id: &str) -> Option { + let sessions = self.0.io().list_sessions(None).ok()?; + let session = sessions.into_iter().find(|s| s.id == id)?; + Some((session.last_activity(), None)) + } + + fn derive(&self, artifact: &ArtifactRef) -> Result { + derive::derive_opencode_session_with(self.0, &artifact.id, false) + } +} + +// ── cursor ───────────────────────────────────────────────────────── + +struct CursorSource<'a>(&'a toolpath_cursor::CursorConvo); + +impl ArtifactSource for CursorSource<'_> { + /// Composer headers (one `SELECT` plus a per-composer bubble-count + /// check) — `lastUpdatedAt` is the fingerprint. Bubble-less drafts + /// are skipped; unlike `share`, composers without a workspace are + /// included, since sync doesn't need to rank them by project. + fn enumerate(&self) -> Vec { + let mut out = Vec::new(); + let listings = match self.0.io().list_composers() { + Ok(l) => l, + Err(e) if is_not_found_cursor(&e) => return out, + Err(e) => { + eprintln!("warning: cursor enumeration failed: {e}"); + return out; + } + }; + for l in listings.into_iter().filter(|l| l.has_bubbles) { + out.push(ArtifactRef { + artifact_type: ArtifactType::Cursor, + modified: l.head.last_updated_at_utc(), + path: l + .head + .workspace_path() + .map(|p| p.to_string_lossy().into_owned()), + id: l.head.composer_id, + size: None, + }); + } + out + } + + fn stamp(&self, _project: Option<&str>, id: &str) -> Option { + let headers = self.0.io().read_composer_headers().ok()?; + let composer = headers + .all_composers + .into_iter() + .find(|c| c.composer_id == id)?; + Some((composer.last_updated_at_utc(), None)) + } + + fn derive(&self, artifact: &ArtifactRef) -> Result { + derive::derive_cursor_session_with(self.0, &artifact.id) + } +} + +// ── pi ───────────────────────────────────────────────────────────── + +struct PiSource<'a>(&'a toolpath_pi::PiConvo); + +impl ArtifactSource for PiSource<'_> { + /// Session files stat-only; the id comes from a one-line header + /// peek, falling back to the filename stem's `_` + /// shape — the same resolution `read_session` accepts. + fn enumerate(&self) -> Vec { + let mut out = Vec::new(); + let projects = match self.0.list_projects() { + Ok(ps) => ps, + Err(e) if is_not_found_pi(&e) => return out, + Err(e) => { + eprintln!("warning: pi enumeration failed: {e}"); + return out; + } + }; + for project in projects { + let files = match toolpath_pi::reader::list_session_files(self.0.resolver(), &project) { + Ok(f) => f, + Err(e) => { + eprintln!("warning: pi project {project} failed: {e}"); + continue; + } + }; + for file in files { + let header_id = toolpath_pi::reader::peek_header(&file) + .ok() + .map(|h| h.id) + .filter(|id| !id.is_empty()); + let stem_id = file + .file_stem() + .and_then(|s| s.to_str()) + .and_then(|s| s.split_once('_')) + .map(|(_, rest)| rest.to_string()); + let Some(id) = header_id.or(stem_id) else { + continue; + }; + let (modified, size) = stat_stamp(&file); + out.push(ArtifactRef { + artifact_type: ArtifactType::Pi, + id, + path: Some(project.clone()), + modified, + size, + }); + } + } + out + } + + fn stamp(&self, project: Option<&str>, id: &str) -> Option { + let files = toolpath_pi::reader::list_session_files(self.0.resolver(), project?).ok()?; + let file = files.into_iter().find(|f| { + toolpath_pi::reader::peek_header(f).is_ok_and(|h| h.id == id) + || f.file_stem() + .and_then(|stem| stem.to_str()) + .and_then(|stem| stem.split_once('_')) + .is_some_and(|(_, rest)| rest == id) + })?; + Some(stat_stamp(&file)) + } + + fn derive(&self, artifact: &ArtifactRef) -> Result { + derive::derive_pi_session_with(self.0, require_path(artifact)?, &artifact.id) + } +} + +// ── copilot ──────────────────────────────────────────────────────── + +struct CopilotSource<'a>(&'a toolpath_copilot::CopilotConvo); + +impl ArtifactSource for CopilotSource<'_> { + /// Session-state directories, stat-only: each session is a + /// `/events.jsonl` under `session-state/` (or its legacy + /// sibling); the directory name is the id and the events file is + /// the fingerprint target. + fn enumerate(&self) -> Vec { + let mut out = Vec::new(); + let mut seen = std::collections::HashSet::new(); + let dirs = [ + self.0.resolver().session_state_dir(), + self.0.resolver().legacy_session_state_dir(), + ]; + for dir in dirs.into_iter().flatten() { + let Ok(entries) = std::fs::read_dir(&dir) else { + continue; + }; + for entry in entries.flatten() { + let Some(id) = entry.file_name().to_str().map(String::from) else { + continue; + }; + let events = entry.path().join("events.jsonl"); + if !events.exists() || !seen.insert(id.clone()) { + continue; + } + let (modified, size) = stat_stamp(&events); + out.push(ArtifactRef { + artifact_type: ArtifactType::Copilot, + id, + path: None, + modified, + size, + }); + } + } + out + } + + fn stamp(&self, _project: Option<&str>, id: &str) -> Option { + let file = self.0.resolver().events_file(id).ok()?; + Some(stat_stamp(&file)) + } + + fn derive(&self, artifact: &ArtifactRef) -> Result { + derive::derive_copilot_session_with(self.0, &artifact.id) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn source_for_missing_provider_or_git_is_none() { + let empty = HarnessBundle::default(); + assert!(source_for(&empty, ArtifactType::Claude).is_none()); + assert!(source_for(&empty, ArtifactType::Git).is_none()); + + let with_claude = HarnessBundle { + claude: Some(toolpath_claude::ClaudeConvo::new()), + ..Default::default() + }; + assert!(source_for(&with_claude, ArtifactType::Claude).is_some()); + assert!( + source_for(&with_claude, ArtifactType::Git).is_none(), + "git is recorded by `p import`, never enumerated or derived by sync" + ); + } +} diff --git a/crates/path-cli/tests/integration.rs b/crates/path-cli/tests/integration.rs index 9f2709fe..85c4f710 100644 --- a/crates/path-cli/tests/integration.rs +++ b/crates/path-cli/tests/integration.rs @@ -685,6 +685,279 @@ fn claude_home_fixture() -> (tempfile::TempDir, PathBuf) { (temp, session_file) } +#[test] +fn cache_sync_ingests_reskips_and_updates() { + let (home, session_file) = claude_home_fixture(); + let cfg = tempfile::tempdir().unwrap(); + let sync = || { + let mut c = cmd(); + c.env("HOME", home.path()) + .env("TOOLPATH_CONFIG_DIR", cfg.path()) + .args(["p", "cache", "sync", "claude"]); + c + }; + + // First run derives the session into the cache and records it. + sync() + .assert() + .success() + .stderr(predicate::str::contains("1 new, 0 updated, 0 unchanged")); + let manifest: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(cfg.path().join("manifest.json")).unwrap()) + .unwrap(); + let record = &manifest["claude"]["session-abc"]; + let cache_id = record["cache_id"].as_str().unwrap(); + assert!( + cfg.path() + .join(format!("documents/{cache_id}.json")) + .exists() + ); + + // Nothing changed: the second run derives nothing. + sync() + .assert() + .success() + .stderr(predicate::str::contains("0 new, 0 updated, 1 unchanged")); + + // The session grows a turn; the third run re-derives it. + let mut body = std::fs::read_to_string(&session_file).unwrap(); + body.push_str( + r#"{"type":"user","uuid":"u-2","timestamp":"2024-01-01T00:05:00Z","cwd":"/x","message":{"role":"user","content":"more"}}"#, + ); + body.push('\n'); + std::fs::write(&session_file, body).unwrap(); + sync() + .assert() + .success() + .stderr(predicate::str::contains("0 new, 1 updated, 0 unchanged")); +} + +#[test] +fn cache_sync_default_run_with_no_sessions_reports_nothing() { + let home = tempfile::tempdir().unwrap(); + let cfg = tempfile::tempdir().unwrap(); + cmd() + .env("HOME", home.path()) + // opencode resolves through $XDG_DATA_HOME before $HOME — drop it + // so the sandboxed run can't see the developer's real database. + .env_remove("XDG_DATA_HOME") + .env("TOOLPATH_CONFIG_DIR", cfg.path()) + .args(["p", "cache", "sync"]) + .assert() + .success() + .stderr(predicate::str::contains("nothing to sync")); + assert!(!cfg.path().join("manifest.json").exists()); +} + +#[test] +fn import_records_manifest_so_sync_skips() { + let (home, _session_file) = claude_home_fixture(); + let cfg = tempfile::tempdir().unwrap(); + let project = home.path().join("proj"); + + cmd() + .env("HOME", home.path()) + .env("TOOLPATH_CONFIG_DIR", cfg.path()) + .args([ + "p", + "import", + "claude", + "--session", + "session-abc", + "--project", + ]) + .arg(&project) + .assert() + .success(); + + let manifest: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(cfg.path().join("manifest.json")).unwrap()) + .unwrap(); + let record = &manifest["claude"]["session-abc"]; + assert!(record["modified"].is_string(), "import must stamp mtime"); + assert!(record["size"].is_u64(), "import must stamp size"); + + // Sync sees the import's record and derives nothing. + cmd() + .env("HOME", home.path()) + .env("TOOLPATH_CONFIG_DIR", cfg.path()) + .args(["p", "cache", "sync", "claude"]) + .assert() + .success() + .stderr(predicate::str::contains("0 new, 0 updated, 1 unchanged")); +} + +#[test] +fn bulk_import_records_every_session() { + let (home, session_file) = claude_home_fixture(); + let cfg = tempfile::tempdir().unwrap(); + let project = home.path().join("proj"); + // A second session alongside the fixture's, so --all has a batch. + std::fs::write( + session_file.parent().unwrap().join("deadbeef-second.jsonl"), + format!( + r#"{{"type":"user","uuid":"u-9","timestamp":"2024-02-01T00:00:00Z","cwd":"{cwd}","message":{{"role":"user","content":"second"}}}} +"#, + cwd = project.display() + ), + ) + .unwrap(); + + cmd() + .env("HOME", home.path()) + .env("TOOLPATH_CONFIG_DIR", cfg.path()) + .args(["p", "import", "claude", "--all", "--project"]) + .arg(&project) + .assert() + .success(); + + let manifest: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(cfg.path().join("manifest.json")).unwrap()) + .unwrap(); + assert_eq!( + manifest["claude"].as_object().unwrap().len(), + 2, + "--all must record every session it writes" + ); + + cmd() + .env("HOME", home.path()) + .env("TOOLPATH_CONFIG_DIR", cfg.path()) + .args(["p", "cache", "sync", "claude"]) + .assert() + .success() + .stderr(predicate::str::contains("0 new, 0 updated, 2 unchanged")); +} + +#[test] +fn git_import_lands_in_manifest_and_sync_leaves_it_alone() { + let (dir, branch) = git_fixture(); + let cfg = tempfile::tempdir().unwrap(); + + cmd() + .env("TOOLPATH_CONFIG_DIR", cfg.path()) + .args(["p", "import", "git", "--branch"]) + .arg(&branch) + .arg("--repo") + .arg(dir.path()) + .assert() + .success(); + + let manifest: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(cfg.path().join("manifest.json")).unwrap()) + .unwrap(); + let git_records = manifest["git"].as_object().unwrap(); + assert_eq!(git_records.len(), 1); + let record = git_records.values().next().unwrap(); + assert!( + record["path"].is_string(), + "git records carry the repo path" + ); + + // Git artifacts are recorded, not discovered: sync reports zeros + // and must not fail or re-derive. + let home = tempfile::tempdir().unwrap(); + cmd() + .env("HOME", home.path()) + .env("TOOLPATH_CONFIG_DIR", cfg.path()) + .args(["p", "cache", "sync", "git"]) + .assert() + .success() + .stderr(predicate::str::contains("0 new, 0 updated, 0 unchanged")); +} + +#[test] +fn share_records_manifest_so_sync_skips() { + let (port, server, _temp, project, home) = share_anon_fixture(); + let cfg = tempfile::tempdir().unwrap(); + + cmd() + .env("HOME", &home) + .env("TOOLPATH_CONFIG_DIR", cfg.path()) + .args([ + "share", + "--harness", + "claude", + "--session", + "session-abc", + "--project", + ]) + .arg(&project) + .args(["--anon", "--url"]) + .arg(format!("http://127.0.0.1:{port}")) + .assert() + .success(); + server.join().unwrap(); + + let manifest: serde_json::Value = + serde_json::from_str(&std::fs::read_to_string(cfg.path().join("manifest.json")).unwrap()) + .unwrap(); + assert!(manifest["claude"]["session-abc"]["modified"].is_string()); + + cmd() + .env("HOME", &home) + .env("TOOLPATH_CONFIG_DIR", cfg.path()) + .args(["p", "cache", "sync", "claude"]) + .assert() + .success() + .stderr(predicate::str::contains("0 new, 0 updated, 1 unchanged")); +} + +#[test] +fn share_uploads_cached_doc_when_source_unchanged() { + let (home, session_file) = claude_home_fixture(); + let cfg = tempfile::tempdir().unwrap(); + let project = home.path().join("proj"); + let share = |port: u16| { + let mut c = cmd(); + c.env("HOME", home.path()) + .env("TOOLPATH_CONFIG_DIR", cfg.path()) + .args([ + "share", + "--harness", + "claude", + "--session", + "session-abc", + "--project", + ]) + .arg(&project) + .args(["--anon", "--url"]) + .arg(format!("http://127.0.0.1:{port}")); + c + }; + + // First share derives + records. + let (port, server) = one_shot_anon_server(); + share(port) + .assert() + .success() + .stderr(predicate::str::contains("uploading without re-deriving").not()); + server.join().unwrap(); + + // Unchanged source: the second share must not re-derive. + let (port, server) = one_shot_anon_server(); + share(port) + .assert() + .success() + .stderr(predicate::str::contains("uploading without re-deriving")); + server.join().unwrap(); + + // The session grows: the fast path steps aside and share re-derives. + let mut body = std::fs::read_to_string(&session_file).unwrap(); + body.push_str( + r#"{"type":"user","uuid":"u-2","timestamp":"2024-01-01T00:05:00Z","cwd":"/x","message":{"role":"user","content":"more"}}"#, + ); + body.push('\n'); + std::fs::write(&session_file, body).unwrap(); + let (port, server) = one_shot_anon_server(); + share(port) + .assert() + .success() + .stderr(predicate::str::contains("uploading without re-deriving").not()) + .stderr(predicate::str::contains("Cached claude session")); + server.join().unwrap(); +} + #[test] fn import_ingests_thinking_maximally() { let (home, session_file) = claude_home_fixture(); @@ -744,6 +1017,17 @@ fn bulk_import_skips_unreadable_sessions() { .stderr(predicate::str::contains("Imported")); } +#[test] +fn cache_sync_rejects_unknown_type() { + let cfg = tempfile::tempdir().unwrap(); + cmd() + .env("TOOLPATH_CONFIG_DIR", cfg.path()) + .args(["p", "cache", "sync", "frobnicate"]) + .assert() + .failure() + .stderr(predicate::str::contains("invalid value")); +} + #[test] fn export_pathbase_repo_flag_requires_login() { // `export pathbase` without --repo falls through to the anonymous diff --git a/crates/toolpath-codex/src/lib.rs b/crates/toolpath-codex/src/lib.rs index e15cf78c..13711fec 100644 --- a/crates/toolpath-codex/src/lib.rs +++ b/crates/toolpath-codex/src/lib.rs @@ -9,7 +9,7 @@ pub mod types; pub use error::{ConvoError, Result}; pub use io::ConvoIO; -pub use paths::PathResolver; +pub use paths::{PathResolver, session_id_from_stem}; pub use reader::RolloutReader; pub use types::{ BaseInstructions, ContentPart, CustomToolCall, CustomToolCallOutput, EventMsg, ExecCommandEnd, diff --git a/crates/toolpath-codex/src/paths.rs b/crates/toolpath-codex/src/paths.rs index 7ca50b51..610b2638 100644 --- a/crates/toolpath-codex/src/paths.rs +++ b/crates/toolpath-codex/src/paths.rs @@ -161,6 +161,55 @@ impl PathResolver { } } +/// The session id embedded in a rollout filename stem +/// (`rollout--`): the trailing UUID when present, +/// otherwise the whole stem. This is the same fallback identity the +/// reader assigns a session whose file has no `session_meta` line, so +/// ids derived from filenames agree with `Session.id` — and either +/// form resolves through [`PathResolver::find_rollout_file`]. +pub fn session_id_from_stem(stem: &str) -> &str { + match find_uuid_start(stem) { + Some(at) => &stem[at..], + None => stem, + } +} + +/// Position of the first 36-char run matching a UUID shape +/// (8-4-4-4-12 hex groups) in the stem. +fn find_uuid_start(stem: &str) -> Option { + let mut idx = 0usize; + let bytes = stem.as_bytes(); + while idx + 36 <= bytes.len() { + if is_uuid_shape(&stem[idx..idx + 36]) { + return Some(idx); + } + idx += 1; + } + None +} + +fn is_uuid_shape(s: &str) -> bool { + let b = s.as_bytes(); + if b.len() != 36 { + return false; + } + for (i, c) in b.iter().enumerate() { + match i { + 8 | 13 | 18 | 23 => { + if *c != b'-' { + return false; + } + } + _ => { + if !c.is_ascii_hexdigit() { + return false; + } + } + } + } + true +} + /// Recursively collect `rollout-*.jsonl` files under `root`. fn walk_for_rollouts(dir: &Path, out: &mut Vec) -> Result<()> { for entry in fs::read_dir(dir)?.flatten() { @@ -296,6 +345,28 @@ mod tests { assert!(p.exists()); } + #[test] + fn is_uuid_shape_accepts_v7() { + assert!(is_uuid_shape("019dabc6-8fef-7681-a054-b5bb75fcb97d")); + assert!(!is_uuid_shape("019dabc6-8fef-7681-a054-b5bb75fcb97")); // too short + assert!(!is_uuid_shape("zzz")); + } + + #[test] + fn session_id_from_stem_extracts_uuid_or_passes_through() { + assert_eq!( + session_id_from_stem( + "rollout-2026-04-20T12-43-30-019dabc6-8fef-7681-a054-b5bb75fcb97d" + ), + "019dabc6-8fef-7681-a054-b5bb75fcb97d" + ); + assert_eq!( + session_id_from_stem("rollout-2026-04-20T10-00-00-abc-xyz"), + "rollout-2026-04-20T10-00-00-abc-xyz" + ); + assert_eq!(session_id_from_stem("not-a-rollout"), "not-a-rollout"); + } + /// A stem whose date doesn't match the directory it sits in misses /// the direct dated-path lookup and must still resolve via the walk. #[test] diff --git a/crates/toolpath-codex/src/reader.rs b/crates/toolpath-codex/src/reader.rs index 6164cef2..4632e164 100644 --- a/crates/toolpath-codex/src/reader.rs +++ b/crates/toolpath-codex/src/reader.rs @@ -105,56 +105,12 @@ impl RolloutReader { } // Fall back to the UUID suffix of the filename stem. if let Some(stem) = path.file_stem().and_then(|s| s.to_str()) { - // Filename pattern: rollout-YYYY-MM-DDThh-mm-ss- - if let Some(uuid_start) = find_uuid_start(stem) { - return stem[uuid_start..].to_string(); - } - return stem.to_string(); + return crate::paths::session_id_from_stem(stem).to_string(); } "unknown".to_string() } } -/// Heuristic: look for the first hex group matching a UUIDv7 shape -/// (8-4-4-4-12 or a prefix thereof) in the filename stem. -fn find_uuid_start(stem: &str) -> Option { - // `rollout-` + `YYYY-MM-DDTHH-MM-SS-` prefix has exactly 28 - // characters before the UUID in normal filenames. - // Fall back to searching for a group of 8 hex characters followed - // by a `-` and more hex. - let mut idx = 0usize; - let bytes = stem.as_bytes(); - while idx + 36 <= bytes.len() { - if is_uuid_shape(&stem[idx..idx + 36]) { - return Some(idx); - } - idx += 1; - } - None -} - -fn is_uuid_shape(s: &str) -> bool { - let b = s.as_bytes(); - if b.len() != 36 { - return false; - } - for (i, c) in b.iter().enumerate() { - match i { - 8 | 13 | 18 | 23 => { - if *c != b'-' { - return false; - } - } - _ => { - if !c.is_ascii_hexdigit() { - return false; - } - } - } - } - true -} - /// Type alias exposed for consumers to avoid re-importing `PathBuf`. pub type RolloutPath = PathBuf; @@ -268,13 +224,6 @@ mod tests { assert!(size > 0); } - #[test] - fn is_uuid_shape_accepts_v7() { - assert!(is_uuid_shape("019dabc6-8fef-7681-a054-b5bb75fcb97d")); - assert!(!is_uuid_shape("019dabc6-8fef-7681-a054-b5bb75fcb97")); // too short - assert!(!is_uuid_shape("zzz")); - } - #[test] fn derive_session_id_falls_back_to_stem_uuid() { let body = r#"{"timestamp":"t","type":"event_msg","payload":{"type":"x"}}"#;