Skip to content

Cross-harness context-compaction provenance (kind v1.2.0) #108

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
benbaarber wants to merge 1 commit into
base: ben/items-ir
Choose a base branch
from
Draft

Cross-harness context-compaction provenance (kind v1.2.0) #108

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
258 changes: 258 additions & 0 deletions CHANGELOG.md
View file
Open in desktop

Large diffs are not rendered by default.

31 changes: 17 additions & 14 deletions CLAUDE.md
View file
Open in desktop

Large diffs are not rendered by default.

14 changes: 10 additions & 4 deletions RFC.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -296,10 +296,16 @@ unrecognized URIs should be treated as a generic path. Kind URIs are
immutable, semver-versioned, and revisions ship at a new version URI.

Defined kinds are listed at <https://toolpath.net/kinds/>. The only one defined
so far is `https://toolpath.net/kinds/agent-coding-session/v1.0.0` — a path
recording an AI coding conversation, where each conversational-turn step
carries a `"conversation.append"` structural change with the turn's role,
text, and so on. See the linked spec for the full contract.
so far is `agent-coding-session`, currently at
`https://toolpath.net/kinds/agent-coding-session/v1.2.0` — a path recording an
AI coding conversation, where each conversational-turn step carries a
`"conversation.append"` structural change with the turn's role, text, and so
on, and context-compaction boundaries carry a `"conversation.compact"` step
whose `kept` list names the prior turns that survived into the post-compaction
window — always a contiguous parent-chain run, oldest first, anchored at its
first element. The earlier `v1.1.0` URI (message-level token accounting,
without `conversation.compact`) and `v1.0.0` URI both remain valid and
documented. See the linked spec for the full contract.

#### Actor Definitions

Expand Down
289 changes: 289 additions & 0 deletions crates/path-cli/kinds/agent-coding-session/v1.2.0/schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,289 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://toolpath.net/kinds/agent-coding-session/v1.2.0/schema.json",
"title": "Toolpath kind: agent-coding-session v1.2.0",
"description": "Additive constraints on a Toolpath `Path` whose `meta.kind` is the agent-coding-session v1.2.0 URI. Apply alongside the base Toolpath schema; the path is valid when both pass. These constraints are by structural `type`, not by artifact key — a `change` entry is checked only when its `structural.type` matches one this kind defines (`conversation.append`, `file.write`, `conversation.event`, `conversation.compact`). Everything is additive: unmentioned properties are allowed, so producer-specific extras never invalidate a path. New in v1.1.0: `group_id` on the turn payload, and the group-level accounting rule — within a `group_id` group, the last step (document order) carries the group's total `token_usage` verbatim from the source and the other steps carry none; values are per-group amounts (never cumulative session counters); summing `token_usage` over a path's steps therefore yields the session totals. The once-per-group rule is normative prose (JSON Schema cannot express it); producers must enforce it. New in v1.2.0: the `conversation.compact` step type, a context-compaction boundary recorded as its own step between the turns it separates. The human-readable contract is at https://toolpath.net/kinds/agent-coding-session/v1.2.0/.",
"type": "object",

"$defs": {
"tokenUsage": {
"type": "object",
"description": "Token accounting for one source group (a message for Claude, a round for Codex). `token_usage` always means the total for a group, verbatim from the source — never a cumulative session counter, never a per-step share. Within a `group_id` group the total sits on the group's last step (document order) and the other steps carry none; a step without a `group_id` is a group of one, so its value is that group's total. Summing over a path's steps therefore yields the session totals. `input_tokens`/`output_tokens` are always emitted (possibly null); cache counters appear only when the source records them.",
"properties": {
"input_tokens": { "type": ["integer", "null"] },
"output_tokens": { "type": ["integer", "null"] },
"cache_read_tokens": { "type": "integer" },
"cache_write_tokens": { "type": "integer" },
"breakdowns": {
"type": "object",
"description": "Optional decomposition of a top-level class into named sub-classes, keyed by the class being broken down (e.g. \"output\"); each value maps sub-class → tokens. INFORMATIONAL: never summed into the total — the parent class already counts these. Invariant: Σ(inner) ≤ the parent class's value.",
"additionalProperties": { "type": "object", "additionalProperties": { "type": "integer" } }
}
},
"required": ["input_tokens", "output_tokens"]
},

"attributedTokenUsage": {
"type": "object",
"description": "This step's own attributed spend, when the source provides step-aligned data — distinct from `token_usage` (the group total). Optional and orthogonal: it rides its own key so summing `token_usage` over steps is unaffected. Within a `group_id` group, the sum of `attributed_token_usage` over its steps is the group's attributed spend; the unattributed remainder (`group token_usage − Σ attributed`) is computed by consumers, never recorded. Same field shape as `tokenUsage`. A producer populates it only when the source genuinely reports per-step spend — among current producers, Codex does (each step is a separate API call with a reported per-call delta); Claude does not (its per-block usage is a cumulative streaming snapshot, not a per-block cost), so Claude-derived steps carry the group total only.",
"properties": {
"input_tokens": { "type": ["integer", "null"] },
"output_tokens": { "type": ["integer", "null"] },
"cache_read_tokens": { "type": "integer" },
"cache_write_tokens": { "type": "integer" }
}
},

"toolResult": {
"type": "object",
"properties": {
"content": { "type": "string" },
"is_error": { "type": "boolean" }
},
"required": ["content", "is_error"]
},

"toolUse": {
"type": "object",
"description": "One tool invocation. `input` is producer-specific JSON (left unconstrained). `category` is Toolpath's classification, or null when the tool is unrecognized. `result` is present only when the result was available in the same turn.",
"properties": {
"id": { "type": "string" },
"name": { "type": "string" },
"input": true,
"category": {
"type": ["string", "null"],
"enum": [
"file_read",
"file_write",
"file_search",
"shell",
"network",
"delegation",
null
]
},
"result": { "$ref": "#/$defs/toolResult" }
},
"required": ["id", "name", "input", "category"]
},

"environment": {
"type": "object",
"description": "Working environment captured at the turn. All fields optional.",
"properties": {
"working_dir": { "type": "string" },
"vcs_branch": { "type": "string" },
"vcs_revision": { "type": "string" }
}
},

"delegation": {
"type": "object",
"description": "Sub-agent work spawned from a turn. `turns` carries the sub-agent's own turns when the producer inlines them.",
"properties": {
"agent_id": { "type": "string" },
"prompt": { "type": "string" },
"turns": { "type": "array" },
"result": { "type": "string" }
},
"required": ["agent_id", "prompt"]
},

"conversationAppend": {
"type": "object",
"description": "The turn payload: the `structural` object of the one `change` entry whose `type` is `conversation.append`. `role` and `text` are always present (text may be empty); everything else appears only when the turn carries it. `group_id` is the provider's identifier for the source accounting unit this turn was derived from — a message for Claude (`message.id`), a round for Codex (`turn_id`). A grouping key, not a step identifier: steps sharing a `group_id` came from one accounting unit (Claude Code writes one JSONL line per content block; a Codex round emits several turns).",
"properties": {
"type": { "const": "conversation.append" },
"role": { "type": "string" },
"text": { "type": "string" },
"thinking": { "type": "string" },
"group_id": { "type": "string" },
"tool_uses": {
"type": "array",
"items": { "$ref": "#/$defs/toolUse" }
},
"token_usage": { "$ref": "#/$defs/tokenUsage" },
"attributed_token_usage": { "$ref": "#/$defs/attributedTokenUsage" },
"stop_reason": { "type": "string" },
"delegations": {
"type": "array",
"items": { "$ref": "#/$defs/delegation" }
},
"environment": { "$ref": "#/$defs/environment" },
"source_parent": {
"type": ["string", "null"],
"description": "The pre-splice source parent (null for a source root), recorded when derive-splicing rewired the step's `parents` through intervening event steps — lets consumers restore the original source linkage exactly."
}
},
"required": ["type", "role", "text"]
},

"fileWrite": {
"type": "object",
"description": "The `structural` object of a sibling `file.write` change keyed by file path. The unified diff (when present) lives on the artifact change's `raw`, not here. `tool_id`/`tool` link the mutation to the `ToolInvocation` that caused it when attributable.",
"properties": {
"type": { "const": "file.write" },
"tool_id": { "type": "string" },
"tool": { "type": "string" },
"operation": { "type": "string" },
"before": { "type": "string" },
"after": { "type": "string" },
"rename_to": { "type": "string" }
},
"required": ["type"]
},

"conversationEvent": {
"type": "object",
"description": "The `structural` object of a `conversation.event` change — a non-turn entry (attachment, preamble line, snapshot, …) preserved for round-trip fidelity. `entry_type` names the source entry kind; the producer's flattened event data rides alongside.",
"properties": {
"type": { "const": "conversation.event" },
"entry_type": { "type": "string" },
"event_source_id": { "type": "string" }
},
"required": ["type", "entry_type"]
},

"conversationCompact": {
"type": "object",
"description": "The `structural` object of a `conversation.compact` change — a context-compaction boundary. Sits as its own step between the turns it separates; only `type` is required and every other field appears only when the producer has it.",
"properties": {
"type": { "const": "conversation.compact" },
"trigger": { "type": "string", "enum": ["auto", "manual"] },
"summary": { "type": "string" },
"pre_tokens": { "type": "number" },
"kept": {
"type": "array",
"description": "Ids of the prior turns that survive verbatim into the post-compaction context window — always a contiguous parent-chain run, oldest first, whose first element is the anchor (the oldest surviving turn). Empty = wholesale (the summary replaced everything). Compaction provenance is this closed field set; native detail richer than the contiguous run is not carried.",
"items": { "type": "string" }
},
"source_parent": {
"type": ["string", "null"],
"description": "The pre-splice source parent (null for a source root), recorded when derive-splicing rewired the step's `parents` through intervening event steps — lets consumers restore the original source linkage exactly."
}
},
"required": ["type"]
},

"artifactChange": {
"type": "object",
"description": "An artifact change, constrained only when its `structural.type` is one this kind defines.",
"allOf": [
{
"if": {
"type": "object",
"properties": {
"structural": {
"type": "object",
"properties": { "type": { "const": "conversation.append" } },
"required": ["type"]
}
},
"required": ["structural"]
},
"then": {
"properties": {
"structural": { "$ref": "#/$defs/conversationAppend" }
}
}
},
{
"if": {
"type": "object",
"properties": {
"structural": {
"type": "object",
"properties": { "type": { "const": "file.write" } },
"required": ["type"]
}
},
"required": ["structural"]
},
"then": {
"properties": {
"structural": { "$ref": "#/$defs/fileWrite" }
}
}
},
{
"if": {
"type": "object",
"properties": {
"structural": {
"type": "object",
"properties": { "type": { "const": "conversation.event" } },
"required": ["type"]
}
},
"required": ["structural"]
},
"then": {
"properties": {
"structural": { "$ref": "#/$defs/conversationEvent" }
}
}
},
{
"if": {
"type": "object",
"properties": {
"structural": {
"type": "object",
"properties": { "type": { "const": "conversation.compact" } },
"required": ["type"]
}
},
"required": ["structural"]
},
"then": {
"properties": {
"structural": { "$ref": "#/$defs/conversationCompact" }
}
}
}
]
}
},

"properties": {
"meta": {
"type": "object",
"description": "Path metadata. `kind` pins this spec; `source` names the producing harness; `producer`/`files_changed`/`vcs_remote` are flattened session-level fields (PathMeta carries `extra` via serde flatten, so they sit directly under `meta`, not under `meta.extra`).",
"properties": {
"kind": {
"const": "https://toolpath.net/kinds/agent-coding-session/v1.2.0"
},
"source": { "type": "string" },
"files_changed": {
"type": "array",
"items": { "type": "string" }
},
"vcs_remote": { "type": "string" },
"producer": {
"type": "object",
"properties": {
"name": { "type": "string" },
"version": { "type": "string" }
},
"required": ["name"]
}
},
"required": ["kind"]
},

"steps": {
"type": "array",
"items": {
"type": "object",
"properties": {
"change": {
"type": "object",
"additionalProperties": { "$ref": "#/$defs/artifactChange" }
}
}
}
}
},

"required": ["meta"]
}
43 changes: 43 additions & 0 deletions crates/path-cli/src/cmd_import.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -220,11 +220,54 @@ fn emit(docs: &[DerivedDoc], force: bool, no_cache: bool, pretty: bool) -> Resul
println!("{}", path.display());
let summary = doc_summary(&d.doc);
eprintln!("Imported {} → {}", summary, d.cache_id);
if let Some(events) = event_type_summary(&d.doc) {
eprintln!(" events preserved: {}", events);
}
}
}
Ok(())
}

/// One-line count of non-conversation events in the derived document,
/// grouped by type — e.g. `attachment ×6, turn_duration ×2`. This is how a
/// source event type our mapping doesn't model stays VISIBLE at import
/// time instead of riding along silently (Copilot's
/// `session.compaction_start`/`complete` pair did exactly that before its
/// encoding was mapped: sessions imported "clean" with `compactions=0`).
fn event_type_summary(doc: &Graph) -> Option<String> {
let mut counts: std::collections::BTreeMap<String, usize> = Default::default();
let paths = doc.paths.iter().filter_map(|p| match p {
toolpath::v1::PathOrRef::Path(path) => Some(path.as_ref()),
toolpath::v1::PathOrRef::Ref(_) => None,
});
for path in paths {
for step in &path.steps {
for ch in step.change.values() {
let Some(s) = &ch.structural else { continue };
if s.change_type != "conversation.event" {
continue;
}
let ty = s
.extra
.get("entry_type")
.and_then(|v| v.as_str())
.unwrap_or("unknown");
*counts.entry(ty.to_string()).or_default() += 1;
}
}
}
if counts.is_empty() {
return None;
}
Some(
counts
.iter()
.map(|(t, n)| format!("{t} ×{n}"))
.collect::<Vec<_>>()
.join(", "),
)
}

fn doc_summary(doc: &Graph) -> String {
if let Some(p) = doc.single_path() {
format!("graph {} (1 path, {} steps)", doc.graph.id, p.steps.len())
Expand Down
2 changes: 1 addition & 1 deletion crates/path-cli/src/cmd_kind.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ mod tests {
#[test]
fn newest_version_resolves_and_is_json() {
let k = kinds::resolve("agent-coding-session").unwrap();
assert_eq!(k.version, "v1.1.0");
assert_eq!(k.version, "v1.2.0");
let _: serde_json::Value =
serde_json::from_str(k.schema).expect("bundled schema is valid JSON");
}
Expand Down
Loading
Loading