diff --git a/README.md b/README.md index 4a07c5b..8ddd9ac 100644 --- a/README.md +++ b/README.md @@ -86,6 +86,8 @@ behind it, and any flags — and writes the proposal to `.tracegrad/`. Nothing touches your prompt until `tracegrad apply`. `apply --revert` restores the snapshot taken before the write. +Harness loop (Claude Code / Pi): [`skills/tracegrad-harness/`](skills/tracegrad-harness/). + Two more commands exist for staged use: `tracegrad attribute` runs the paid attribution pass alone and caches it, and `tracegrad propose` then produces the proposal for the cost of a single synthesis call. `tracegrad trends` compares diff --git a/skills/tracegrad-harness/README.md b/skills/tracegrad-harness/README.md new file mode 100644 index 0000000..76b189f --- /dev/null +++ b/skills/tracegrad-harness/README.md @@ -0,0 +1,35 @@ +# Tracegrad harness skill pack + +Harness loop for Claude Code / Pi. Core stays lean. Sidecars live beside the user repo, never under `src/tracegrad/`. No Kitaru in this pack. + +`tracegrad run` does not write the prompt. Harness apply is only `tracegrad apply --accept ` after a human or policy names those indices — procedure in [review-edits](review-edits/SKILL.md). Never invent accepts. + +CLI flags: `tracegrad --help`. Manifest, `.tracegradrc`, and project state: repo README. + +## Loop + +1. [import-traces](import-traces/SKILL.md) — adapt-in a user-named export → JSONL +2. [propose-edits](propose-edits/SKILL.md) — estimate, then run (or attribute + propose); cards on disk +3. [review-edits](review-edits/SKILL.md) — show cards; `--accept` after human- or policy-named indices +4. [export-prompt](export-prompt/SKILL.md) — adapt-out after apply wrote the template +5. [next-batch](next-batch/SKILL.md) — conductor over 1–4, then `status` / `trends` + +## Reach + +| Situation | Skill | +| --- | --- | +| import traces, adapt-in, JSONL from a store | [import-traces](import-traces/SKILL.md) | +| estimate, propose, run, attribute | [propose-edits](propose-edits/SKILL.md) | +| cards, review, accept, apply, policy | [review-edits](review-edits/SKILL.md) | +| export / adapt-out the applied prompt | [export-prompt](export-prompt/SKILL.md) | +| next batch, close the loop | [next-batch](next-batch/SKILL.md) | + +JSONL ingest rules: [import-traces/jsonl-contract.md](import-traces/jsonl-contract.md). + +## Sidecars and policy + +Copy [examples/](examples/) beside the user repo: + +- `sidecar-adapt-in.py` — `FIELD_MAP` = Tracegrad field → foreign path +- `sidecar-adapt-out.py` — applied template → user path +- `policy.commented.toml` — agent-side apply gate (unattended apply off by default; `accept` is a TOML integer array). Core does not load this file. diff --git a/skills/tracegrad-harness/examples/policy.commented.toml b/skills/tracegrad-harness/examples/policy.commented.toml new file mode 100644 index 0000000..bcfa18c --- /dev/null +++ b/skills/tracegrad-harness/examples/policy.commented.toml @@ -0,0 +1,38 @@ +# Tracegrad harness apply policy — EXAMPLE +# +# Agent-side gate for review-edits (next-batch follows that skill). +# NOT loaded by Tracegrad core. Core still only writes the prompt via +# `tracegrad apply --accept `. +# +# Copy to the project (suggested name: tracegrad-apply-policy.toml) and +# edit. Unattended apply stays OFF until a human sets unattended_apply +# and names accept indices. Missing file, false, or ambiguous → stop +# and ask. A human may still name indices for attended --accept. +# Never invent --accept values. There is no allow_all. + +# Fully unattended apply is OFF by default. Leave this false (or omit it). +unattended_apply = false + +# Card indices for `tracegrad apply --accept`. Empty / omitted / commented +# means do not apply. The agent must not guess, rank, or "take the safe ones". +# Example after a human named cards 0 and 2: +# accept = [0, 2] +accept = [] + +# DELETE edits: refuse unless a human explicitly sets this true. +allow_delete = false + +# Instruction ids the agent must not apply, even if they survived core gates. +# Complements `.tracegradrc` neverDelete (which core already enforces). +# Example: neverDelete = ["prompt/identity"] +neverDelete = [] + +# If the proposal's tokens_after would exceed this, stop and ask. +# Uncomment and set a positive integer to enable. +# token_ceiling = 4000 + +# Optional: pin a run. If omitted, the skill uses the latest proposal. +# run_id = "run-0001" + +# Do not set a flag that means "accept every card". There is no allow_all +# here on purpose. `tracegrad apply --all` is not a policy default. diff --git a/skills/tracegrad-harness/examples/sidecar-adapt-in.py b/skills/tracegrad-harness/examples/sidecar-adapt-in.py new file mode 100644 index 0000000..57c8bb5 --- /dev/null +++ b/skills/tracegrad-harness/examples/sidecar-adapt-in.py @@ -0,0 +1,136 @@ +#!/usr/bin/env python3 +"""Sidecar adapt-in: map a foreign trace export to Tracegrad JSONL. + +Copy this file next to the user repo. Do not move it into src/tracegrad/. +The user pipeline stays unchanged; this adapter is the only place that +learns the foreign field names. This stub is not a vendor integration. + +One JSON object per line; ingest forbids extra keys. Required: trace_id, +input, output, judge.score in [0, 1], judge.rationale, prompt_hash. +Optional meta.model. Ingest drops duplicate trace_ids, rationales under +24 usable characters, and non-dominant prompt_hash. + +FIELD_MAP (or --map-json) is Tracegrad field → foreign dotted path. +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path +from typing import Any, Mapping + +# Tracegrad field → foreign dotted path. Edit per project. +FIELD_MAP: dict[str, str] = { + "trace_id": "id", + "input": "prompt", + "output": "response", + "judge.score": "score", + "judge.rationale": "rationale", + "prompt_hash": "prompt_hash", + "meta.model": "model", +} + + +def _get(record: Mapping[str, Any], path: str) -> Any: + current: Any = record + for part in path.split("."): + if not isinstance(current, Mapping) or part not in current: + return None + current = current[part] + return current + + +def adapt_record(record: Mapping[str, Any], field_map: Mapping[str, str]) -> dict[str, Any]: + """Map one foreign object to a Tracegrad trace dict. Drop incomplete rows.""" + + def mapped(key: str) -> Any: + source = field_map.get(key) + if not source: + return None + return _get(record, source) + + trace_id = mapped("trace_id") + rationale = mapped("judge.rationale") + prompt_hash = mapped("prompt_hash") + score = mapped("judge.score") + if trace_id is None or rationale is None or prompt_hash is None or score is None: + return {} + try: + score_f = float(score) + except (TypeError, ValueError): + return {} + if not 0.0 <= score_f <= 1.0: + return {} + + out: dict[str, Any] = { + "trace_id": str(trace_id), + "input": "" if mapped("input") is None else str(mapped("input")), + "output": "" if mapped("output") is None else str(mapped("output")), + "judge": {"score": score_f, "rationale": str(rationale)}, + "prompt_hash": str(prompt_hash), + } + model = mapped("meta.model") + if model: + out["meta"] = {"model": str(model)} + return out + + +def iter_records(path: Path) -> list[Mapping[str, Any]]: + text = path.read_text(encoding="utf-8") + stripped = text.lstrip() + if stripped.startswith("["): + payload = json.loads(text) + if not isinstance(payload, list): + raise SystemExit(f"{path}: expected a JSON array") + return [row for row in payload if isinstance(row, Mapping)] + rows: list[Mapping[str, Any]] = [] + for line_number, line in enumerate(text.splitlines(), start=1): + if not line.strip(): + continue + row = json.loads(line) + if not isinstance(row, Mapping): + raise SystemExit(f"{path}:{line_number}: expected a JSON object") + rows.append(row) + return rows + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--source", required=True, help="foreign JSON or JSONL export") + parser.add_argument("--out", required=True, help="Tracegrad JSONL destination") + parser.add_argument( + "--map-json", + default=None, + help="optional JSON object overriding FIELD_MAP", + ) + args = parser.parse_args(argv) + + field_map = dict(FIELD_MAP) + if args.map_json: + override = json.loads(Path(args.map_json).read_text(encoding="utf-8")) + if not isinstance(override, dict): + print("tracegrad sidecar: --map-json must be an object", file=sys.stderr) + return 1 + field_map.update({str(k): str(v) for k, v in override.items()}) + + written = 0 + skipped = 0 + destination = Path(args.out) + destination.parent.mkdir(parents=True, exist_ok=True) + with destination.open("w", encoding="utf-8") as handle: + for record in iter_records(Path(args.source)): + adapted = adapt_record(record, field_map) + if not adapted: + skipped += 1 + continue + handle.write(json.dumps(adapted, ensure_ascii=False) + "\n") + written += 1 + + print(f"wrote {written} traces to {destination} ({skipped} skipped)", file=sys.stderr) + return 0 if written else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/skills/tracegrad-harness/examples/sidecar-adapt-out.py b/skills/tracegrad-harness/examples/sidecar-adapt-out.py new file mode 100644 index 0000000..63dd81c --- /dev/null +++ b/skills/tracegrad-harness/examples/sidecar-adapt-out.py @@ -0,0 +1,53 @@ +#!/usr/bin/env python3 +"""Sidecar adapt-out: copy the applied Tracegrad template to the user path. + +Run only after `tracegrad apply` has written the manifest template. +Copy this file next to the user repo. Do not move it into src/tracegrad/. +This adapter does not apply edits and does not invent a destination. + +If --from and --to are the same path, this is a no-op. That is the usual +case when the app already loads the manifest `template_file`. +""" + +from __future__ import annotations + +import argparse +import shutil +import sys +from pathlib import Path + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--from", + dest="source", + required=True, + help="applied template (manifest template_file, resolved)", + ) + parser.add_argument( + "--to", + dest="destination", + required=True, + help="path the user pipeline actually loads", + ) + args = parser.parse_args(argv) + + source = Path(args.source) + destination = Path(args.destination) + if not source.is_file(): + print(f"tracegrad sidecar: applied template not found: {source}", file=sys.stderr) + print("apply first; this adapter does not write the prompt itself", file=sys.stderr) + return 1 + if source.resolve() == destination.resolve(): + print(f"adapt-out no-op: {source} is already the user path", file=sys.stderr) + return 0 + + destination.parent.mkdir(parents=True, exist_ok=True) + shutil.copyfile(source, destination) + print(f"copied {source} -> {destination}", file=sys.stderr) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/skills/tracegrad-harness/export-prompt/SKILL.md b/skills/tracegrad-harness/export-prompt/SKILL.md new file mode 100644 index 0000000..db32302 --- /dev/null +++ b/skills/tracegrad-harness/export-prompt/SKILL.md @@ -0,0 +1,36 @@ +--- +name: export-prompt +description: "Adapt-out the applied template to a user-named path. Invoke for export after apply. Not apply." +--- + +# Export prompt + +Sidecar adapt-out: copy the template `tracegrad apply` already wrote to the path the user's app loads. Apply is [`../review-edits/SKILL.md`](../review-edits/SKILL.md). Export is not a Tracegrad subcommand. + +Skip (successful no-op) when the manifest `template_file` **is** the user path. Do not copy a proposed, unapplied template. + +## Steps + +1. **Confirm apply for this run.** Look for a new prompt hash on apply stdout, a new line in `.tracegrad/ledgers/applied.jsonl`, and a snapshot under `.tracegrad/snapshots/`. + + Done: at least one of those exists. If none, stop and send the user to `review-edits`. + +2. **Source.** Resolve manifest `template_file` against the same `--base-directory` used at apply (`tracegrad apply --help`). The source is the path apply printed (`applied … to