From 3200da326155d988926d5604dcfebc24867acd13 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Fri, 4 Sep 2026 00:01:10 +0000 Subject: [PATCH 1/5] docs(skills): scaffold harness skill pack for Claude Code / Pi MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add verb-first skills under skills/tracegrad-harness/ so a harness can drive import → estimate → propose → review → export without bloating core. Unattended apply stays off unless a policy file permits it. Adapters remain sidecar examples, not package code. Co-authored-by: Dickson Neoh --- README.md | 2 + skills/tracegrad-harness/README.md | 73 +++++++++ .../examples/policy.commented.toml | 37 +++++ .../examples/sidecar-adapt-in.py | 143 +++++++++++++++++ .../examples/sidecar-adapt-out.py | 52 +++++++ .../tracegrad-harness/export-prompt/SKILL.md | 105 +++++++++++++ .../tracegrad-harness/import-traces/SKILL.md | 117 ++++++++++++++ skills/tracegrad-harness/next-batch/SKILL.md | 145 +++++++++++++++++ .../tracegrad-harness/propose-edits/SKILL.md | 124 +++++++++++++++ .../tracegrad-harness/review-edits/SKILL.md | 146 ++++++++++++++++++ 10 files changed, 944 insertions(+) create mode 100644 skills/tracegrad-harness/README.md create mode 100644 skills/tracegrad-harness/examples/policy.commented.toml create mode 100644 skills/tracegrad-harness/examples/sidecar-adapt-in.py create mode 100644 skills/tracegrad-harness/examples/sidecar-adapt-out.py create mode 100644 skills/tracegrad-harness/export-prompt/SKILL.md create mode 100644 skills/tracegrad-harness/import-traces/SKILL.md create mode 100644 skills/tracegrad-harness/next-batch/SKILL.md create mode 100644 skills/tracegrad-harness/propose-edits/SKILL.md create mode 100644 skills/tracegrad-harness/review-edits/SKILL.md diff --git a/README.md b/README.md index 4a07c5b..8ddd9ac 100644 --- a/README.md +++ b/README.md @@ -86,6 +86,8 @@ behind it, and any flags — and writes the proposal to `.tracegrad/`. Nothing touches your prompt until `tracegrad apply`. `apply --revert` restores the snapshot taken before the write. +Harness loop (Claude Code / Pi): [`skills/tracegrad-harness/`](skills/tracegrad-harness/). + Two more commands exist for staged use: `tracegrad attribute` runs the paid attribution pass alone and caches it, and `tracegrad propose` then produces the proposal for the cost of a single synthesis call. `tracegrad trends` compares diff --git a/skills/tracegrad-harness/README.md b/skills/tracegrad-harness/README.md new file mode 100644 index 0000000..a0ec9d0 --- /dev/null +++ b/skills/tracegrad-harness/README.md @@ -0,0 +1,73 @@ +# Tracegrad harness skill pack + +A verb-first skill pack so Claude Code or Pi can drive Tracegrad from +**outside** the Python package. Core stays lean and harness-driven. This +directory is documentation and examples, not a product surface and not a +dependency. + +Kitaru is not in core. Do not add Kitaru code, deps, or docs from this pack. +Adapters live beside the user's repo (sidecar), never under `src/tracegrad/`. + +## The loop + +1. **Import traces.** A sidecar adapt-in maps the user's existing store into + Tracegrad JSONL. Their pipeline does not change. Tracegrad never learns the + stack. +2. **Propose edits.** `tracegrad run --estimate`, then `tracegrad run`. Analysis + writes cards and a proposal under `.tracegrad/`. It does **not** write the + prompt. +3. **Review edits.** Default: stop and ask the human. Apply only when a local + policy file is present **and** clearly permits it. Missing or ambiguous + policy → stop and ask. Never invent `--accept` indices. +4. **Export prompt.** After a successful `tracegrad apply`, a sidecar adapt-out + copies the written template back to the path the user's app actually loads + (no-op if that path already *is* the manifest `template_file`). +5. **Next batch.** Thin conductor: import → estimate → propose → review → + export → `tracegrad status` / `tracegrad trends`. Unattended apply stays + off unless the policy file both exists and permits it. + +The existing gate is unchanged: `tracegrad run` never writes the prompt; only +`tracegrad apply` does, and only after a human or an explicit policy accept. + +## Skills + +| Skill | Folder | Does | +| --- | --- | --- | +| import traces | [`import-traces/`](import-traces/SKILL.md) | Adapt-in → JSONL | +| propose edits | [`propose-edits/`](propose-edits/SKILL.md) | Estimate + run (or attribute + propose); stop with cards | +| review edits | [`review-edits/`](review-edits/SKILL.md) | Show cards; apply only if policy allows | +| export prompt | [`export-prompt/`](export-prompt/SKILL.md) | Adapt-out after apply | +| next batch | [`next-batch/`](next-batch/SKILL.md) | Conduct the loop; stop at review unless policy permits apply | + +## Policy file + +The harness agent may read a project-local policy file (suggested name: +`tracegrad-apply-policy.toml` at the project root, or a path the user names). +**Tracegrad core does not load this file.** `.tracegradrc` remains the only +config core reads (`neverDelete`, coverage, harness presets). The policy file +is an extra, agent-side gate on whether the skill may invoke `tracegrad apply`. + +Shape (see [`examples/policy.commented.toml`](examples/policy.commented.toml)): + +- `unattended_apply` — default **false**. Off means stop and ask. +- `accept` — comma-style list of card indices for `tracegrad apply --accept`. + Empty, omitted, or guessed → do not apply. +- `allow_delete` — default **false**. Skip or refuse `DELETE` edits. +- `neverDelete` — instruction ids the agent must not apply, even if they + survived core gates. Complements `.tracegradrc`; does not replace it. +- `token_ceiling` — if the proposal's `tokens_after` would exceed this, stop + and ask rather than apply. + +If the file is missing, `unattended_apply` is false, `accept` is empty, or any +rule is ambiguous: **stop and ask**. Never pass `--all` unless the policy +explicitly lists every index a human already named. + +## Adapters stay outside core + +Copy the stubs in [`examples/`](examples/) next to the user's repo: + +- `sidecar-adapt-in.py` — foreign traces → JSONL contract +- `sidecar-adapt-out.py` — applied template → user path + +Do not vendor these into `src/tracegrad/`. Do not add a Kitaru (or any other +eval-stack) integration to the package to make import/export "just work". diff --git a/skills/tracegrad-harness/examples/policy.commented.toml b/skills/tracegrad-harness/examples/policy.commented.toml new file mode 100644 index 0000000..f16ce8e --- /dev/null +++ b/skills/tracegrad-harness/examples/policy.commented.toml @@ -0,0 +1,37 @@ +# Tracegrad harness apply policy — EXAMPLE +# +# Read by Claude Code / Pi skill pack (review-edits, next-batch). +# NOT loaded by Tracegrad core. Core still only writes the prompt via +# `tracegrad apply`, and only for the indices passed to `--accept`. +# +# Copy to the project (suggested name: tracegrad-apply-policy.toml) and +# edit. Unattended apply stays OFF until a human sets unattended_apply +# and names accept indices. Missing file, false, or ambiguous → the +# skill stops and asks. Never invent --accept values. + +# Fully unattended apply is OFF by default. Leave this false (or omit it). +unattended_apply = false + +# Card indices for `tracegrad apply --accept`. Empty / omitted / commented +# means do not apply. The agent must not guess, rank, or "take the safe ones". +# Example after a human named cards 0 and 2: +# accept = [0, 2] +accept = [] + +# DELETE edits: refuse unless a human explicitly sets this true. +allow_delete = false + +# Instruction ids the agent must not apply, even if they survived core gates. +# Complements `.tracegradrc` neverDelete (which core already enforces). +# Example: neverDelete = ["prompt/identity"] +neverDelete = [] + +# If the proposal's tokens_after would exceed this, stop and ask. +# Uncomment and set a positive integer to enable. +# token_ceiling = 4000 + +# Optional: pin a run. If omitted, the skill uses the latest proposal. +# run_id = "run-0001" + +# Do not set a flag that means "accept every card". There is no allow_all +# here on purpose. `tracegrad apply --all` is not a policy default. diff --git a/skills/tracegrad-harness/examples/sidecar-adapt-in.py b/skills/tracegrad-harness/examples/sidecar-adapt-in.py new file mode 100644 index 0000000..6b6ecbb --- /dev/null +++ b/skills/tracegrad-harness/examples/sidecar-adapt-in.py @@ -0,0 +1,143 @@ +#!/usr/bin/env python3 +"""Sidecar adapt-in: map a foreign trace export to Tracegrad JSONL. + +Copy this file next to the user repo. Do not move it into src/tracegrad/. +The user pipeline stays unchanged; this adapter is the only place that +learns the foreign field names. + +JSONL contract (one object per line, extra keys forbidden by ingest): + + { + "trace_id": str, + "input": str, + "output": str, + "judge": {"score": float in [0, 1], "rationale": str}, + "prompt_hash": str, + "meta": {"model": str} # optional + } + +Fill FIELD_MAP (or --map-json) for the local export. This stub is not a +vendor integration. +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path +from typing import Any, Mapping + +# Foreign dotted paths → Tracegrad fields. Edit per project. +FIELD_MAP: dict[str, str] = { + "trace_id": "id", + "input": "prompt", + "output": "response", + "judge.score": "score", + "judge.rationale": "rationale", + "prompt_hash": "prompt_hash", + "meta.model": "model", +} + + +def _get(record: Mapping[str, Any], path: str) -> Any: + current: Any = record + for part in path.split("."): + if not isinstance(current, Mapping) or part not in current: + return None + current = current[part] + return current + + +def adapt_record(record: Mapping[str, Any], field_map: Mapping[str, str]) -> dict[str, Any]: + """Map one foreign object to a Tracegrad trace dict. Drop incomplete rows.""" + + def mapped(key: str) -> Any: + source = field_map.get(key) + if not source: + return None + return _get(record, source) + + trace_id = mapped("trace_id") + rationale = mapped("judge.rationale") + prompt_hash = mapped("prompt_hash") + score = mapped("judge.score") + if trace_id is None or rationale is None or prompt_hash is None or score is None: + return {} + try: + score_f = float(score) + except (TypeError, ValueError): + return {} + if not 0.0 <= score_f <= 1.0: + return {} + + out: dict[str, Any] = { + "trace_id": str(trace_id), + "input": "" if mapped("input") is None else str(mapped("input")), + "output": "" if mapped("output") is None else str(mapped("output")), + "judge": {"score": score_f, "rationale": str(rationale)}, + "prompt_hash": str(prompt_hash), + } + model = mapped("meta.model") + if model: + out["meta"] = {"model": str(model)} + return out + + +def iter_records(path: Path) -> list[Mapping[str, Any]]: + text = path.read_text(encoding="utf-8") + stripped = text.lstrip() + if stripped.startswith("["): + payload = json.loads(text) + if not isinstance(payload, list): + raise SystemExit(f"{path}: expected a JSON array") + return [row for row in payload if isinstance(row, Mapping)] + rows: list[Mapping[str, Any]] = [] + for line_number, line in enumerate(text.splitlines(), start=1): + if not line.strip(): + continue + row = json.loads(line) + if not isinstance(row, Mapping): + raise SystemExit(f"{path}:{line_number}: expected a JSON object") + rows.append(row) + return rows + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--source", required=True, help="foreign JSON or JSONL export") + parser.add_argument("--out", required=True, help="Tracegrad JSONL destination") + parser.add_argument( + "--map-json", + default=None, + help="optional JSON object overriding FIELD_MAP", + ) + args = parser.parse_args(argv) + + field_map = dict(FIELD_MAP) + if args.map_json: + override = json.loads(Path(args.map_json).read_text(encoding="utf-8")) + if not isinstance(override, dict): + print("tracegrad sidecar: --map-json must be an object", file=sys.stderr) + return 1 + field_map.update({str(k): str(v) for k, v in override.items()}) + + written = 0 + skipped = 0 + destination = Path(args.out) + destination.parent.mkdir(parents=True, exist_ok=True) + with destination.open("w", encoding="utf-8") as handle: + for record in iter_records(Path(args.source)): + adapted = adapt_record(record, field_map) + if not adapted: + skipped += 1 + continue + handle.write(json.dumps(adapted, ensure_ascii=False) + "\n") + written += 1 + + print(f"wrote {written} traces to {destination} ({skipped} skipped)", file=sys.stderr) + return 0 if written else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/skills/tracegrad-harness/examples/sidecar-adapt-out.py b/skills/tracegrad-harness/examples/sidecar-adapt-out.py new file mode 100644 index 0000000..6d0306a --- /dev/null +++ b/skills/tracegrad-harness/examples/sidecar-adapt-out.py @@ -0,0 +1,52 @@ +#!/usr/bin/env python3 +"""Sidecar adapt-out: copy the applied Tracegrad template to the user path. + +Run only after `tracegrad apply` has written the manifest template. +Copy this file next to the user repo. Do not move it into src/tracegrad/. + +If --from and --to are the same path, this is a no-op. That is the usual +case when the app already loads the manifest `template_file`. +""" + +from __future__ import annotations + +import argparse +import shutil +import sys +from pathlib import Path + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--from", + dest="source", + required=True, + help="applied template (manifest template_file, resolved)", + ) + parser.add_argument( + "--to", + dest="destination", + required=True, + help="path the user pipeline actually loads", + ) + args = parser.parse_args(argv) + + source = Path(args.source) + destination = Path(args.destination) + if not source.is_file(): + print(f"tracegrad sidecar: applied template not found: {source}", file=sys.stderr) + print("apply first; this adapter does not write the prompt itself", file=sys.stderr) + return 1 + if source.resolve() == destination.resolve(): + print(f"adapt-out no-op: {source} is already the user path", file=sys.stderr) + return 0 + + destination.parent.mkdir(parents=True, exist_ok=True) + shutil.copyfile(source, destination) + print(f"copied {source} -> {destination}", file=sys.stderr) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/skills/tracegrad-harness/export-prompt/SKILL.md b/skills/tracegrad-harness/export-prompt/SKILL.md new file mode 100644 index 0000000..96aaa3b --- /dev/null +++ b/skills/tracegrad-harness/export-prompt/SKILL.md @@ -0,0 +1,105 @@ +--- +name: export-prompt +description: After a successful tracegrad apply, copy the written template back to the user path via a sidecar adapt-out. Use when asked to export, sync, or deploy the applied prompt. Does not apply. +--- + +# Export prompt + +Sidecar adapt-out: after `tracegrad apply` has already written the template, +copy or write that file to the path the user's app actually loads. + +This skill does **not** apply edits. If apply has not happened, stop and send +the user to `review-edits`. + +## When to use + +- Apply succeeded and the production prompt is a *different* path than the + manifest `template_file`. +- The user asks to export, sync, or copy the applied prompt back. +- `next-batch` reaches the export step after a permitted apply. + +Skip (successful no-op) when the manifest path **is** the user path. Do not +use this skill to bypass the apply gate by writing a "proposed" prompt. + +## Sidecar adapt-out + +Tracegrad writes the file named in the manifest, resolved against +`--base-directory`. The adapter lives beside the user repo. Copy +[`../examples/sidecar-adapt-out.py`](../examples/sidecar-adapt-out.py) and +point `--from` at the applied template and `--to` at the user path. + +```sh +python sidecar-adapt-out.py \ + --from path/from/manifest/prompt.md \ + --to /path/the/user/app/loads/prompt.md +``` + +Do not add an export command to Tracegrad core. Do not vendor the user's +prompt store into `src/tracegrad/`. + +## Steps + +1. Confirm apply already happened for this run: new prompt hash on stdout, or + an entry in `.tracegrad/ledgers/applied.jsonl`, plus a snapshot under + `.tracegrad/snapshots/`. If none, **stop** — export has nothing safe to + copy. + +2. Resolve the source path: manifest `template_file` + `--base-directory`. + +3. Resolve the destination: only a path the user named (config, flag, or + existing sidecar defaults). Do not guess a production path. + +4. Run the sidecar. Overwrite only that destination. + +5. Report source, destination, and that core was not modified. + +## Inputs / outputs + +**Inputs** + +- Applied template path (manifest `template_file`). +- User destination path. +- Sidecar script (example: `examples/sidecar-adapt-out.py`). +- Optional: `--project-root` / `--base-directory` used during apply, so the + same file is found. + +**Outputs** + +- User-path file updated to match the applied template (or no-op if identical + path / identical bytes). +- No Tracegrad state writes. No second apply. + +## Failure modes + +| Failure | What to do | +| --- | --- | +| No apply yet | Stop. Do not copy a pre-apply template and call it exported. | +| Stale proposal was refused | Nothing to export. Re-run propose + review. | +| Destination unknown | Stop and ask. | +| Destination outside what the user named | Refuse. | +| Apply succeeded but template hash does not match apply output | Stop; do not overwrite the user path with a file that may have been edited out of band. | +| Urge to `tracegrad apply` "so there is something to export" | Refuse unless `review-edits` policy/human already allowed it. | + +## Exact CLI + +Export is not a Tracegrad subcommand. Apply (already done, other skill): + +```sh +tracegrad apply --accept 0,2 +``` + +Verify after export, if useful: + +```sh +tracegrad status --manifest manifest.json +``` + +Do not run: + +```sh +tracegrad run +tracegrad apply --all +tracegrad apply --revert +``` + +from this skill. diff --git a/skills/tracegrad-harness/import-traces/SKILL.md b/skills/tracegrad-harness/import-traces/SKILL.md new file mode 100644 index 0000000..2728c05 --- /dev/null +++ b/skills/tracegrad-harness/import-traces/SKILL.md @@ -0,0 +1,117 @@ +--- +name: import-traces +description: Pull the latest batch from the user store into Tracegrad JSONL via a sidecar adapt-in. Use when asked to import traces, prepare a batch, or feed Tracegrad without changing the user pipeline. +--- + +# Import traces + +Map the user's existing trace store into Tracegrad JSONL. The user pipeline +stays unchanged. The adapter lives **beside** their repo, not in Tracegrad +core. Do not add vendor integrations under `src/tracegrad/`. + +## When to use + +- A new eval or production batch is ready and needs to enter Tracegrad. +- The user asks to import, adapt-in, or "pull latest traces". +- `next-batch` needs a JSONL path before `tracegrad run`. + +Do not use this skill to call `tracegrad apply`, edit the prompt, or invent +fields the store does not have. + +## JSONL contract + +One JSON object per line. Extra keys are rejected (`extra: forbid`). Required: + +```json +{ + "trace_id": "t-001", + "input": "user or task text", + "output": "model response", + "judge": {"score": 0.4, "rationale": "why this score, in words"}, + "prompt_hash": "sha256:… of the prompt version that produced this trace" +} +``` + +| Field | Rules | +| --- | --- | +| `trace_id` | Non-empty string. Unique within the file. | +| `input` / `output` | Strings (may be empty, but output is what violations quote). | +| `judge.score` | Number in `[0.0, 1.0]`. | +| `judge.rationale` | Non-empty string. Ingest drops rationales shorter than 24 usable characters (`rationale-below-quality-floor`). A score without a real rationale is not a batch. | +| `prompt_hash` | Non-empty string identifying the prompt version. Mixed hashes: only the dominant partition is kept. | +| `meta` | Optional. Only `meta.model` is allowed. Mixed models are reported, not dropped. | + +Do not emit `null` for required strings. Do not put the judge score in a +different shape (`label`, `pass`, nested vendor blobs). Map those in the +sidecar. + +A manifest is not part of the JSONL. It is a separate JSON file +(`template_file`, `engine`, `judge_fingerprint`, …) passed to `tracegrad run +--manifest`. Confirm it exists; do not rewrite the user's stack to produce it. + +## Thin adapt-in + +1. Copy [`../examples/sidecar-adapt-in.py`](../examples/sidecar-adapt-in.py) next + to the user's repo (or edit their existing adapter). +2. Fill the field map: vendor id → `trace_id`, prompt/response → `input`/`output`, + judge score + rationale, prompt version → `prompt_hash`. +3. Write JSONL. Leave Tracegrad unaware of the source format. + +```sh +python sidecar-adapt-in.py --source /path/to/user-export --out batch.jsonl +``` + +## Steps + +1. Confirm `tracegrad init` has been run in the project (creates `.tracegrad/`): + + ```sh + tracegrad init + ``` + +2. Locate the latest user-store export. Do not scrape production APIs unless + the user named the source. +3. Run the sidecar adapt-in. Do not import traces by hand-writing JSONL unless + the batch is tiny and the user asked. +4. Sanity-check: line count, unique `trace_id`, rationale length, single + dominant `prompt_hash`. +5. Hand the JSONL path to `propose-edits` / `next-batch`. Stop. This skill + does not run analysis. + +## Inputs / outputs + +**Inputs** + +- Path to the user store or export (file, directory, or command the user named). +- Sidecar script path (default: a copy of `examples/sidecar-adapt-in.py`). +- Destination JSONL path (for example `batch.jsonl`). +- Optional: `--project-root` if `.tracegrad/` is not cwd. + +**Outputs** + +- JSONL file matching the contract above. +- A short note: how many lines written, which source, which `prompt_hash` values + were seen. No prompt writes. + +## Failure modes + +| Failure | What to do | +| --- | --- | +| No adapter and unknown store format | Stop and ask. Do not invent a vendor integration in core. | +| Missing rationale or score not in `[0, 1]` | Fix the map or drop the row in the adapter; do not pad fake rationales. | +| Duplicate `trace_id` | Deduplicate in the adapter; ingest will drop later copies as `duplicate-trace-id`. | +| Several `prompt_hash` values | Warn: ingest keeps only the dominant partition. Split batches if the user wants both versions. | +| Empty export | Stop. Do not call `tracegrad run` on an empty file. | +| Temptation to "just add a connector to src/tracegrad" | Refuse. Sidecar only. | + +## Exact CLI + +```sh +tracegrad init +# Adapt-in is not a tracegrad subcommand. Then, later: +tracegrad run --traces batch.jsonl --manifest manifest.json --estimate +``` + +`tracegrad init` is the only Tracegrad command this skill should run. Import +does not call `tracegrad run`, `attribute`, `propose`, `apply`, `status`, or +`trends`. diff --git a/skills/tracegrad-harness/next-batch/SKILL.md b/skills/tracegrad-harness/next-batch/SKILL.md new file mode 100644 index 0000000..7369a44 --- /dev/null +++ b/skills/tracegrad-harness/next-batch/SKILL.md @@ -0,0 +1,145 @@ +--- +name: next-batch +description: Conduct one Tracegrad loop — import, estimate, propose, review, maybe export, then status/trends. Unattended apply only if a policy file exists and permits it; otherwise stop at review. +--- + +# Next batch + +Thin conductor over the other skills. Follow them; do not invent a parallel +pipeline. Unattended apply is **off** unless a policy file is present **and** +permits it. Otherwise stop at review and ask. + +`tracegrad run` never writes the prompt. Only `tracegrad apply` does, and only +after human or policy accept. + +## When to use + +- The user asks to run the next batch, close the loop, or "do a Tracegrad + pass". +- A new export is ready and they want import through trends in one go. + +Do not use when they only asked for one step (import, estimate, review). +Invoke that skill instead. + +## Steps + +1. **Import** — follow [`../import-traces/SKILL.md`](../import-traces/SKILL.md). + Sidecar adapt-in → JSONL. User pipeline unchanged. + + ```sh + tracegrad init + python sidecar-adapt-in.py --source /path/to/user-export --out batch.jsonl + ``` + +2. **Estimate** — follow [`../propose-edits/SKILL.md`](../propose-edits/SKILL.md). + + ```sh + tracegrad run --traces batch.jsonl --manifest manifest.json --estimate + ``` + + If spend is unclear, stop and ask before the paid run. + +3. **Propose** — same skill. Default: + + ```sh + tracegrad run --traces batch.jsonl --manifest manifest.json + ``` + + Staged alternative: + + ```sh + tracegrad attribute --traces batch.jsonl --manifest manifest.json + tracegrad propose --traces batch.jsonl --manifest manifest.json + ``` + + Stop with cards on disk under `.tracegrad/runs//`. Do not apply here. + +4. **Review** — follow [`../review-edits/SKILL.md`](../review-edits/SKILL.md). + + - Policy file missing, `unattended_apply` not true, `accept` empty, or any + rule ambiguous → **stop and ask**. Show the cards. Do not apply. + - Policy present **and** permits a specific `--accept` list → apply only + those indices. + + ```sh + tracegrad apply --accept 0,2 + ``` + + Never invent accepts. Never `--all` as a shortcut. + +5. **Export** — only if apply actually wrote the template. Follow + [`../export-prompt/SKILL.md`](../export-prompt/SKILL.md). + + ```sh + python sidecar-adapt-out.py --from prompt.md --to /user/path/prompt.md + ``` + + Skip if apply was not run, or if the user path is already the manifest file. + +6. **Status / trends** + + ```sh + tracegrad status + tracegrad status --manifest manifest.json + tracegrad trends + ``` + + Trends need at least two runs. Advisory only — never auto-revert. + +## Unattended apply (strict) + +Apply without a human in the loop **only** when all of: + +1. A policy file exists (see [`../examples/policy.commented.toml`](../examples/policy.commented.toml)). +2. `unattended_apply = true` (default in the example is **false**). +3. `accept` names real card indices; the agent does not fill gaps. +4. `allow_delete`, `neverDelete`, and `token_ceiling` all pass. + +Otherwise the conductor **stops at review**. + +## Inputs / outputs + +**Inputs** + +- User-store location, sidecar paths, JSONL destination, manifest, + `--project-root` / `--base-directory`. +- Optional policy file. Absence is a valid input: it means stop at review. + +**Outputs** + +- JSONL batch, `.tracegrad/` proposal (always if run succeeded). +- Applied template + adapt-out copy **only** if review allowed apply. +- Status / trend text. No autonomous revert. + +## Failure modes + +| Failure | What to do | +| --- | --- | +| Import produced empty/invalid JSONL | Stop before `run`. | +| Estimate too large / no model configured | Stop and ask. | +| No edits proposed | Skip apply/export; still run `status` / `trends` if two reports exist. | +| Policy would apply but proposal is stale | Stop. Re-propose; do not `--force`. | +| Review blocked | Leave the prompt untouched. Report cards and wait. | +| Export destination unknown after apply | Prompt was still written by apply; ask where to copy. Do not revert. | +| Second batch with a changed judge fingerprint | Trends may be incomparable; report that instead of forcing a delta. | + +## Exact CLI (full sequence) + +```sh +tracegrad init +# sidecar adapt-in → batch.jsonl +tracegrad run --traces batch.jsonl --manifest manifest.json --estimate +tracegrad run --traces batch.jsonl --manifest manifest.json +# stop here unless policy/human named indices +tracegrad apply --accept 0,2 +# sidecar adapt-out if the user path differs +tracegrad status --manifest manifest.json +tracegrad trends +``` + +Staged variant replaces the paid `run` with: + +```sh +tracegrad attribute --traces batch.jsonl --manifest manifest.json +tracegrad propose --traces batch.jsonl --manifest manifest.json +``` diff --git a/skills/tracegrad-harness/propose-edits/SKILL.md b/skills/tracegrad-harness/propose-edits/SKILL.md new file mode 100644 index 0000000..14db1b4 --- /dev/null +++ b/skills/tracegrad-harness/propose-edits/SKILL.md @@ -0,0 +1,124 @@ +--- +name: propose-edits +description: Estimate then run Tracegrad analysis so edit cards land under .tracegrad/. Use when asked to propose prompt edits, run a batch, or attribute+propose. Never apply. +--- + +# Propose edits + +Run analysis and stop with review cards on disk. **Do not apply.** +`tracegrad run` never writes the prompt; only `tracegrad apply` does. + +## When to use + +- JSONL + manifest are ready and the user wants proposals. +- After `import-traces`, before `review-edits`. +- The user asks to estimate cost, run the pipeline, or use the staged + attribute → propose path. + +Do not use this skill to accept edits, pass `--accept` / `--all`, or copy the +template anywhere. + +## Steps (default path) + +1. Confirm project state exists: + + ```sh + tracegrad init + ``` + +2. Cost preview. No model is contacted: + + ```sh + tracegrad run --traces batch.jsonl --manifest manifest.json --estimate + ``` + + Optional location flags (same for every command below): + + ```sh + tracegrad run --traces batch.jsonl --manifest manifest.json \ + --project-root . --base-directory . --estimate + ``` + +3. If the estimate is acceptable to the user (or they already approved spend), + run the full pipeline: + + ```sh + tracegrad run --traces batch.jsonl --manifest manifest.json + ``` + + Useful flags: + + ```sh + tracegrad run --traces batch.jsonl --manifest manifest.json --jobs 8 + tracegrad run --traces batch.jsonl --manifest manifest.json --token-ceiling 4000 + tracegrad run --traces batch.jsonl --manifest manifest.json --run-id run-0001 + ``` + +4. Stop. Cards print to stdout. The proposal is + `.tracegrad/runs//proposal.json`. Hand off to `review-edits`. + +## Steps (staged path) + +Use when attribution should be paid once and synthesis retried: + +```sh +tracegrad attribute --traces batch.jsonl --manifest manifest.json +tracegrad propose --traces batch.jsonl --manifest manifest.json +``` + +`propose` is `run` against cached attributions (one synthesis call, not +another pass over the batch). It still does not write the prompt. + +`tracegrad propose --traces batch.jsonl --manifest manifest.json --estimate` +is valid and is the same preview as `run --estimate`. + +## Inputs / outputs + +**Inputs** + +- `--traces` — JSONL from `import-traces`. +- `--manifest` — JSON with `template_file`, `engine` (`none` or `format`), + `judge_fingerprint`. +- Optional: `--project-root`, `--base-directory`, `--jobs`, `--token-ceiling`, + `--run-id`, `--session-id`. +- Model access: `.tracegradrc` harness presets, or env key for the openai + provider. Do not silently switch providers. + +**Outputs (on disk under `.tracegrad/`)** + +| Path | What | +| --- | --- | +| `runs//proposal.json` | Proposed edits, diffs, evidence, token counts | +| `runs//` | Resume checkpoint; autopsy of dropped proposals | +| `reports/*.json` | Theme counts for later `trends` | +| `distilled/` | Content-addressed traces quotes must match | +| `ledgers/` | Runs, gaps, rejections (append-only) | + +Stdout from `run` / `propose` already prints one card per surviving edit +(`[index] OPERATION instruction_id`, theme, diff, quotes). "No edits +proposed" is a valid outcome. + +## Failure modes + +| Failure | What to do | +| --- | --- | +| Estimate looks expensive | Stop and ask before `run` without `--estimate`. | +| Missing API key / harness binary | Report the error. Do not disable gates or invent a backend. | +| Ingest drops most traces | Show dropped reasons (`rationale-below-quality-floor`, `invalid-schema`, `prompt-hash-partition`). Fix JSONL; do not lower the floor in core. | +| Stale or missing manifest / template | Stop. `--base-directory` must be where `template_file` resolves. | +| Proposal empty | Valid. Do not force synthesis. Proceed to `status` if asked. | +| Urge to "just apply the obvious ones" | Refuse. That is `review-edits`, and default is stop/ask. | + +## Exact CLI + +```sh +tracegrad init +tracegrad run --traces batch.jsonl --manifest manifest.json --estimate +tracegrad run --traces batch.jsonl --manifest manifest.json +tracegrad attribute --traces batch.jsonl --manifest manifest.json +tracegrad propose --traces batch.jsonl --manifest manifest.json +# Not in this skill: +# tracegrad apply +# tracegrad apply --accept … +# tracegrad apply --all +``` diff --git a/skills/tracegrad-harness/review-edits/SKILL.md b/skills/tracegrad-harness/review-edits/SKILL.md new file mode 100644 index 0000000..03b8235 --- /dev/null +++ b/skills/tracegrad-harness/review-edits/SKILL.md @@ -0,0 +1,146 @@ +--- +name: review-edits +description: Show Tracegrad review cards and stop to ask the human. Apply only when a policy file clearly allows it. Use when asked to review, accept, or apply proposed edits. Never invent --accept indices. +--- + +# Review edits + +Show the cards from the latest proposal. **Default: stop and ask.** Fully +unattended apply is off unless a policy file is present **and** permits it. + +Only `tracegrad apply` writes the prompt. Do not edit the template by hand to +"save a step". Do not pass `--all` unless a human-named policy lists every +index. + +## When to use + +- After `propose-edits`, when cards exist under `.tracegrad/`. +- The user asks to review, accept, reject, or apply. +- `next-batch` reaches the review step. + +Do not use this skill to re-run attribution or to export; those are other +skills. Do not apply if policy is missing or ambiguous. + +## Steps + +1. Find the proposal. Latest run id is the sorted name under + `.tracegrad/runs/*/proposal.json`. Or pass `--run-id` the user named. + +2. Show every card to the human: index, `ADD`/`REWRITE`/`DELETE`, instruction + id, theme, flags, unified diff, verbatim quotes. Prefer the cards already + printed by `tracegrad run`. Re-read `proposal.json` if the terminal scroll + is gone. Do not call `apply --all` in order to "see" the result. + +3. **Default path — stop and ask.** List the indices and wait. A human types + which cards to accept. Then: + + ```sh + tracegrad apply + ``` + + Interactive TTY: one `[y/N]` per card. Non-TTY without `--accept`/`--all`: + nothing is applied (exit 1). That is correct, not a prompt to guess. + + After a human names indices: + + ```sh + tracegrad apply --accept 0,2 + ``` + +4. **Policy path — optional, gated.** Read the project policy file (see + [`../examples/policy.commented.toml`](../examples/policy.commented.toml)). + Apply **only** if every condition holds: + + - File exists and parses. + - `unattended_apply` is explicitly `true`. + - `accept` is a non-empty list of integer indices that exist on this + proposal. **Never invent or extend that list.** + - Each selected edit is allowed: not `DELETE` unless `allow_delete = true`; + instruction id not in policy `neverDelete`; proposal `tokens_after` + does not exceed policy `token_ceiling` when that key is set. + - Indices still match the current proposal (same `run_id`, template hash + not stale). + + Then, and only then: + + ```sh + tracegrad apply --accept 0,2 + ``` + + Use `--run-id` when the policy or user named a run: + + ```sh + tracegrad apply --run-id run-0001 --accept 0,2 + ``` + +5. If any check fails or is ambiguous: **stop and ask**. Do not fall back to + `--all`. Do not apply a subset the policy did not name. + +## Policy vs `.tracegradrc` + +| File | Who reads it | +| --- | --- | +| `.tracegradrc` | Tracegrad core (`neverDelete`, coverage, harness presets) | +| apply policy (this skill) | The harness agent, before it may run `tracegrad apply` | + +Core `neverDelete` already drops protected deletes at synthesize time. The +policy `neverDelete` is a second, agent-side refuse-to-apply list. It does +not replace the rc file. + +## Inputs / outputs + +**Inputs** + +- `.tracegrad/runs//proposal.json` +- Optional policy file (path the user named, else + `tracegrad-apply-policy.toml` in the project root if present) +- Optional `--run-id`, `--project-root`, `--base-directory` + +**Outputs** + +- Review text for the human (always). +- If apply ran: Tracegrad prints accepted count, new prompt hash, snapshot + path under `.tracegrad/snapshots/`. +- If apply did not run: a clear stop/ask message. Prompt file unchanged. + +Revert is a separate, explicit user request: + +```sh +tracegrad apply --revert +tracegrad apply --revert --force +``` + +Do not revert as part of ordinary review. + +## Failure modes + +| Failure | What to do | +| --- | --- | +| No proposal | Tell the user to run `propose-edits` first. `tracegrad apply` exits 1. | +| Policy missing / `unattended_apply` false / omitted | Stop and ask. | +| `accept` empty, omitted, or contains unknown indices | Stop and ask. Never invent. | +| Selected edit is `DELETE` and `allow_delete` is not true | Skip apply; stop and ask. | +| Instruction id in policy `neverDelete` | Do not include it in `--accept`. If that empties the list, stop. | +| `tokens_after` over `token_ceiling` | Stop and ask. | +| "template changed … proposal is stale" | Do not `--force` apply. Re-run `propose-edits`. | +| Non-TTY, no `--accept` | Nothing applied. Ask the human; do not switch to `--all`. | +| `--all` looks convenient | Forbidden unless the policy's `accept` list is exactly every index a human already approved. Prefer `--accept` with that list. | + +## Exact CLI + +```sh +tracegrad apply +tracegrad apply --accept 0,2 +tracegrad apply --run-id run-0001 --accept 0 +tracegrad apply --project-root . --base-directory . --accept 0 +tracegrad apply --revert +# Do not use unless a human-named policy lists every index: +# tracegrad apply --all +``` + +Related, not this skill's job: + +```sh +tracegrad status +tracegrad trends +``` From deff02d17ad9e8671dee051c03ff55607f532b1d Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Fri, 4 Sep 2026 00:08:09 +0000 Subject: [PATCH 2/5] docs(skills): close harness apply and CLI cargo-cult holes Forbid bare interactive apply on the harness path; require --accept with human- or policy-named indices. Distinguish unattended policy apply from attended human accepts. Drop run from import-traces Exact CLI and omit inventable apply indices from next-batch Exact CLI. Co-authored-by: Dickson Neoh --- skills/tracegrad-harness/README.md | 21 ++++--- .../tracegrad-harness/import-traces/SKILL.md | 4 +- skills/tracegrad-harness/next-batch/SKILL.md | 22 +++++--- .../tracegrad-harness/review-edits/SKILL.md | 56 +++++++++++-------- 4 files changed, 62 insertions(+), 41 deletions(-) diff --git a/skills/tracegrad-harness/README.md b/skills/tracegrad-harness/README.md index a0ec9d0..8879894 100644 --- a/skills/tracegrad-harness/README.md +++ b/skills/tracegrad-harness/README.md @@ -16,15 +16,19 @@ Adapters live beside the user's repo (sidecar), never under `src/tracegrad/`. 2. **Propose edits.** `tracegrad run --estimate`, then `tracegrad run`. Analysis writes cards and a proposal under `.tracegrad/`. It does **not** write the prompt. -3. **Review edits.** Default: stop and ask the human. Apply only when a local - policy file is present **and** clearly permits it. Missing or ambiguous - policy → stop and ask. Never invent `--accept` indices. +3. **Review edits.** Default: stop and ask the human. **Unattended** apply + needs a local policy file that is present **and** clearly permits it. + **Attended** apply is allowed after the human explicitly names card + indices (`tracegrad apply --accept …`). Missing or ambiguous policy does + not block that attended path; it only forbids applying with no + human-named list. Never invent `--accept` indices. 4. **Export prompt.** After a successful `tracegrad apply`, a sidecar adapt-out copies the written template back to the path the user's app actually loads (no-op if that path already *is* the manifest `template_file`). 5. **Next batch.** Thin conductor: import → estimate → propose → review → export → `tracegrad status` / `tracegrad trends`. Unattended apply stays - off unless the policy file both exists and permits it. + off unless the policy file both exists and permits it. Attended apply + after explicit human indices is allowed; otherwise stop at review. The existing gate is unchanged: `tracegrad run` never writes the prompt; only `tracegrad apply` does, and only after a human or an explicit policy accept. @@ -35,9 +39,9 @@ The existing gate is unchanged: `tracegrad run` never writes the prompt; only | --- | --- | --- | | import traces | [`import-traces/`](import-traces/SKILL.md) | Adapt-in → JSONL | | propose edits | [`propose-edits/`](propose-edits/SKILL.md) | Estimate + run (or attribute + propose); stop with cards | -| review edits | [`review-edits/`](review-edits/SKILL.md) | Show cards; apply only if policy allows | +| review edits | [`review-edits/`](review-edits/SKILL.md) | Show cards; `--accept` only after human or policy names indices | | export prompt | [`export-prompt/`](export-prompt/SKILL.md) | Adapt-out after apply | -| next batch | [`next-batch/`](next-batch/SKILL.md) | Conduct the loop; stop at review unless policy permits apply | +| next batch | [`next-batch/`](next-batch/SKILL.md) | Conduct the loop; unattended apply needs policy, else stop at review | ## Policy file @@ -59,8 +63,9 @@ Shape (see [`examples/policy.commented.toml`](examples/policy.commented.toml)): and ask rather than apply. If the file is missing, `unattended_apply` is false, `accept` is empty, or any -rule is ambiguous: **stop and ask**. Never pass `--all` unless the policy -explicitly lists every index a human already named. +rule is ambiguous: **do not apply unattended** — stop and ask. A human may +still name indices for attended `--accept`. Never pass `--all` unless the +policy explicitly lists every index a human already named. ## Adapters stay outside core diff --git a/skills/tracegrad-harness/import-traces/SKILL.md b/skills/tracegrad-harness/import-traces/SKILL.md index 2728c05..82ba464 100644 --- a/skills/tracegrad-harness/import-traces/SKILL.md +++ b/skills/tracegrad-harness/import-traces/SKILL.md @@ -108,8 +108,8 @@ python sidecar-adapt-in.py --source /path/to/user-export --out batch.jsonl ```sh tracegrad init -# Adapt-in is not a tracegrad subcommand. Then, later: -tracegrad run --traces batch.jsonl --manifest manifest.json --estimate +# Adapt-in is not a tracegrad subcommand: +python sidecar-adapt-in.py --source /path/to/user-export --out batch.jsonl ``` `tracegrad init` is the only Tracegrad command this skill should run. Import diff --git a/skills/tracegrad-harness/next-batch/SKILL.md b/skills/tracegrad-harness/next-batch/SKILL.md index 7369a44..b51892b 100644 --- a/skills/tracegrad-harness/next-batch/SKILL.md +++ b/skills/tracegrad-harness/next-batch/SKILL.md @@ -7,7 +7,8 @@ description: Conduct one Tracegrad loop — import, estimate, propose, review, m Thin conductor over the other skills. Follow them; do not invent a parallel pipeline. Unattended apply is **off** unless a policy file is present **and** -permits it. Otherwise stop at review and ask. +permits it. Attended apply after explicit human indices is allowed; otherwise +stop at review and ask. `tracegrad run` never writes the prompt. Only `tracegrad apply` does, and only after human or policy accept. @@ -57,12 +58,17 @@ Invoke that skill instead. 4. **Review** — follow [`../review-edits/SKILL.md`](../review-edits/SKILL.md). - Policy file missing, `unattended_apply` not true, `accept` empty, or any - rule ambiguous → **stop and ask**. Show the cards. Do not apply. - - Policy present **and** permits a specific `--accept` list → apply only + rule ambiguous → **stop and ask**. Show the cards. Do not apply + unattended. + - Human names indices in this conversation → attended apply of **only** those indices. + - Policy present **and** permits a specific `--accept` list → unattended + apply of **only** those indices. ```sh - tracegrad apply --accept 0,2 + # Run only after HUMAN_OR_POLICY_INDICES is replaced with supplied integers. + # Do not substitute example numbers. Do not answer interactive [y/N]. + tracegrad apply --accept ``` Never invent accepts. Never `--all` as a shortcut. @@ -130,9 +136,11 @@ tracegrad init # sidecar adapt-in → batch.jsonl tracegrad run --traces batch.jsonl --manifest manifest.json --estimate tracegrad run --traces batch.jsonl --manifest manifest.json -# stop here unless policy/human named indices -tracegrad apply --accept 0,2 -# sidecar adapt-out if the user path differs +# STOP at review. Show cards. Do not apply, and do not invent --accept indices. +# Apply is not part of this canonical sequence. If a human or policy has +# already supplied indices, review-edits may run: +# tracegrad apply --accept +# sidecar adapt-out only after a real apply tracegrad status --manifest manifest.json tracegrad trends ``` diff --git a/skills/tracegrad-harness/review-edits/SKILL.md b/skills/tracegrad-harness/review-edits/SKILL.md index 03b8235..8eeb7a1 100644 --- a/skills/tracegrad-harness/review-edits/SKILL.md +++ b/skills/tracegrad-harness/review-edits/SKILL.md @@ -1,16 +1,19 @@ --- name: review-edits -description: Show Tracegrad review cards and stop to ask the human. Apply only when a policy file clearly allows it. Use when asked to review, accept, or apply proposed edits. Never invent --accept indices. +description: Show Tracegrad review cards and stop to ask the human. Apply only with tracegrad apply --accept after a human names indices, or when a policy file lists them. Never answer interactive y/N. Never invent --accept indices. Never use bare apply. --- # Review edits Show the cards from the latest proposal. **Default: stop and ask.** Fully unattended apply is off unless a policy file is present **and** permits it. +Attended apply is allowed after the human names card indices. -Only `tracegrad apply` writes the prompt. Do not edit the template by hand to -"save a step". Do not pass `--all` unless a human-named policy lists every -index. +Only `tracegrad apply --accept ` writes the prompt on the harness +path. Do not edit the template by hand to "save a step". Do not pass `--all` +unless a human-named policy lists every index. **Do not run bare +`tracegrad apply`.** That form is interactive `[y/N]` per card; a TTY agent +can answer those prompts itself. That is not a human-named accept. ## When to use @@ -19,7 +22,8 @@ index. - `next-batch` reaches the review step. Do not use this skill to re-run attribution or to export; those are other -skills. Do not apply if policy is missing or ambiguous. +skills. Do not apply unattended if policy is missing or ambiguous. Attended +apply still requires explicit human-named `--accept` indices. ## Steps @@ -31,21 +35,21 @@ skills. Do not apply if policy is missing or ambiguous. printed by `tracegrad run`. Re-read `proposal.json` if the terminal scroll is gone. Do not call `apply --all` in order to "see" the result. -3. **Default path — stop and ask.** List the indices and wait. A human types - which cards to accept. Then: +3. **Default path — stop and ask.** List the card indices and wait. Do not + call `tracegrad apply` yet. Do not answer `[y/N]`. A human must type which + cards to accept (for example `0` and `2`). Only after those indices exist + in this conversation, substitute them and run: ```sh - tracegrad apply + tracegrad apply --accept ``` - Interactive TTY: one `[y/N]` per card. Non-TTY without `--accept`/`--all`: - nothing is applied (exit 1). That is correct, not a prompt to guess. - - After a human names indices: - - ```sh - tracegrad apply --accept 0,2 - ``` + The placeholder is not an example list. Replace it with the integers the + human typed (or, on the policy path below, the integers the file listed). + **Forbidden for the harness agent:** bare `tracegrad apply`, answering + interactive `[y/N]`, piping `yes`/`y` into apply, or using `--all` to skip + naming indices. Non-TTY without `--accept`/`--all` applies nothing (exit + 1); that is correct, not a prompt to guess or to switch to a TTY. 4. **Policy path — optional, gated.** Read the project policy file (see [`../examples/policy.commented.toml`](../examples/policy.commented.toml)). @@ -64,13 +68,13 @@ skills. Do not apply if policy is missing or ambiguous. Then, and only then: ```sh - tracegrad apply --accept 0,2 + tracegrad apply --accept ``` Use `--run-id` when the policy or user named a run: ```sh - tracegrad apply --run-id run-0001 --accept 0,2 + tracegrad apply --run-id run-0001 --accept ``` 5. If any check fails or is ambiguous: **stop and ask**. Do not fall back to @@ -117,24 +121,28 @@ Do not revert as part of ordinary review. | Failure | What to do | | --- | --- | | No proposal | Tell the user to run `propose-edits` first. `tracegrad apply` exits 1. | -| Policy missing / `unattended_apply` false / omitted | Stop and ask. | +| Policy missing / `unattended_apply` false / omitted | Do not apply unattended. Stop and ask. Attended `--accept` is allowed if the human named indices. | | `accept` empty, omitted, or contains unknown indices | Stop and ask. Never invent. | | Selected edit is `DELETE` and `allow_delete` is not true | Skip apply; stop and ask. | | Instruction id in policy `neverDelete` | Do not include it in `--accept`. If that empties the list, stop. | | `tokens_after` over `token_ceiling` | Stop and ask. | | "template changed … proposal is stale" | Do not `--force` apply. Re-run `propose-edits`. | | Non-TTY, no `--accept` | Nothing applied. Ask the human; do not switch to `--all`. | +| TTY offers `[y/N]` / bare `tracegrad apply` | Do not answer. Stop and collect indices, then `--accept` only. | | `--all` looks convenient | Forbidden unless the policy's `accept` list is exactly every index a human already approved. Prefer `--accept` with that list. | ## Exact CLI ```sh -tracegrad apply -tracegrad apply --accept 0,2 -tracegrad apply --run-id run-0001 --accept 0 -tracegrad apply --project-root . --base-directory . --accept 0 +# Harness path — only after human-typed or policy-listed indices replace the placeholder: +tracegrad apply --accept +tracegrad apply --run-id run-0001 --accept +tracegrad apply --project-root . --base-directory . --accept +# Explicit user request to revert only: tracegrad apply --revert -# Do not use unless a human-named policy lists every index: +# Forbidden for the harness agent: +# tracegrad apply +# (do not answer interactive [y/N]) # tracegrad apply --all ``` From 397754e7feff9f0e41c8ec46c704575108cdf5ae Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Fri, 4 Sep 2026 00:11:54 +0000 Subject: [PATCH 3/5] docs(skills): close remaining apply cargo-cult and map-direction holes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drop inventable apply indices from export-prompt Exact CLI. Correct sidecar FIELD_MAP comment to Tracegrad field → foreign path. Forbid --all on the harness path; describe policy accept as a TOML integer array. Co-authored-by: Dickson Neoh --- skills/tracegrad-harness/README.md | 9 +++++---- .../examples/sidecar-adapt-in.py | 2 +- skills/tracegrad-harness/export-prompt/SKILL.md | 14 +++++++++----- skills/tracegrad-harness/review-edits/SKILL.md | 17 +++++++++-------- 4 files changed, 24 insertions(+), 18 deletions(-) diff --git a/skills/tracegrad-harness/README.md b/skills/tracegrad-harness/README.md index 8879894..66f1533 100644 --- a/skills/tracegrad-harness/README.md +++ b/skills/tracegrad-harness/README.md @@ -54,8 +54,9 @@ is an extra, agent-side gate on whether the skill may invoke `tracegrad apply`. Shape (see [`examples/policy.commented.toml`](examples/policy.commented.toml)): - `unattended_apply` — default **false**. Off means stop and ask. -- `accept` — comma-style list of card indices for `tracegrad apply --accept`. - Empty, omitted, or guessed → do not apply. +- `accept` — list of integers (TOML array, e.g. `accept = [0, 2]`). The CLI + flag is still comma-separated: `tracegrad apply --accept 0,2`. Empty, + omitted, or guessed → do not apply. - `allow_delete` — default **false**. Skip or refuse `DELETE` edits. - `neverDelete` — instruction ids the agent must not apply, even if they survived core gates. Complements `.tracegradrc`; does not replace it. @@ -64,8 +65,8 @@ Shape (see [`examples/policy.commented.toml`](examples/policy.commented.toml)): If the file is missing, `unattended_apply` is false, `accept` is empty, or any rule is ambiguous: **do not apply unattended** — stop and ask. A human may -still name indices for attended `--accept`. Never pass `--all` unless the -policy explicitly lists every index a human already named. +still name indices for attended `--accept`. Never pass `--all`; always pass +the explicit `--accept` list. ## Adapters stay outside core diff --git a/skills/tracegrad-harness/examples/sidecar-adapt-in.py b/skills/tracegrad-harness/examples/sidecar-adapt-in.py index 6b6ecbb..5588dc8 100644 --- a/skills/tracegrad-harness/examples/sidecar-adapt-in.py +++ b/skills/tracegrad-harness/examples/sidecar-adapt-in.py @@ -28,7 +28,7 @@ from pathlib import Path from typing import Any, Mapping -# Foreign dotted paths → Tracegrad fields. Edit per project. +# Tracegrad field → foreign dotted path. Edit per project. FIELD_MAP: dict[str, str] = { "trace_id": "id", "input": "prompt", diff --git a/skills/tracegrad-harness/export-prompt/SKILL.md b/skills/tracegrad-harness/export-prompt/SKILL.md index 96aaa3b..cca07a6 100644 --- a/skills/tracegrad-harness/export-prompt/SKILL.md +++ b/skills/tracegrad-harness/export-prompt/SKILL.md @@ -82,10 +82,14 @@ prompt store into `src/tracegrad/`. ## Exact CLI -Export is not a Tracegrad subcommand. Apply (already done, other skill): +Export is not a Tracegrad subcommand. Do **not** call `tracegrad apply` here +(that is `review-edits`, and only with `--accept ` +after those indices were supplied). ```sh -tracegrad apply --accept 0,2 +python sidecar-adapt-out.py \ + --from path/from/manifest/prompt.md \ + --to /path/the/user/app/loads/prompt.md ``` Verify after export, if useful: @@ -94,12 +98,12 @@ Verify after export, if useful: tracegrad status --manifest manifest.json ``` -Do not run: +Do not run from this skill: ```sh tracegrad run +tracegrad apply +tracegrad apply --accept tracegrad apply --all tracegrad apply --revert ``` - -from this skill. diff --git a/skills/tracegrad-harness/review-edits/SKILL.md b/skills/tracegrad-harness/review-edits/SKILL.md index 8eeb7a1..0a4fa32 100644 --- a/skills/tracegrad-harness/review-edits/SKILL.md +++ b/skills/tracegrad-harness/review-edits/SKILL.md @@ -1,6 +1,6 @@ --- name: review-edits -description: Show Tracegrad review cards and stop to ask the human. Apply only with tracegrad apply --accept after a human names indices, or when a policy file lists them. Never answer interactive y/N. Never invent --accept indices. Never use bare apply. +description: Show Tracegrad review cards and stop to ask the human. Apply only with tracegrad apply --accept after a human names indices, or when a policy file lists them. Never answer interactive y/N. Never invent --accept indices. Never use bare apply or --all. --- # Review edits @@ -10,10 +10,11 @@ unattended apply is off unless a policy file is present **and** permits it. Attended apply is allowed after the human names card indices. Only `tracegrad apply --accept ` writes the prompt on the harness -path. Do not edit the template by hand to "save a step". Do not pass `--all` -unless a human-named policy lists every index. **Do not run bare -`tracegrad apply`.** That form is interactive `[y/N]` per card; a TTY agent -can answer those prompts itself. That is not a human-named accept. +path. Do not edit the template by hand to "save a step". **Never pass +`--all`.** Always pass the explicit `--accept` list (human-named or +policy-listed). **Do not run bare `tracegrad apply`.** That form is +interactive `[y/N]` per card; a TTY agent can answer those prompts itself. +That is not a human-named accept. ## When to use @@ -77,8 +78,8 @@ apply still requires explicit human-named `--accept` indices. tracegrad apply --run-id run-0001 --accept ``` -5. If any check fails or is ambiguous: **stop and ask**. Do not fall back to - `--all`. Do not apply a subset the policy did not name. +5. If any check fails or is ambiguous: **stop and ask**. Never `--all`. Do + not apply a subset the policy did not name. ## Policy vs `.tracegradrc` @@ -129,7 +130,7 @@ Do not revert as part of ordinary review. | "template changed … proposal is stale" | Do not `--force` apply. Re-run `propose-edits`. | | Non-TTY, no `--accept` | Nothing applied. Ask the human; do not switch to `--all`. | | TTY offers `[y/N]` / bare `tracegrad apply` | Do not answer. Stop and collect indices, then `--accept` only. | -| `--all` looks convenient | Forbidden unless the policy's `accept` list is exactly every index a human already approved. Prefer `--accept` with that list. | +| `--all` looks convenient | Never. Always pass `--accept` with the explicit human- or policy-named list. | ## Exact CLI From 9ce6d4992307f5660e9a39d72d5e9583127ea4b3 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Fri, 4 Sep 2026 05:24:25 +0000 Subject: [PATCH 4/5] docs(skills): rewrite harness pack to the writing-for-agents bar Front-load trigger words, ordered steps with checkable done-conditions, and a single apply-gate in review-edits. Disclose the JSONL contract beside import-traces. Captain locks unchanged: --accept only, unattended apply off by default, no Kitaru, adapters stay sidecars. Co-authored-by: Dickson Neoh --- skills/tracegrad-harness/README.md | 88 +++------- .../examples/policy.commented.toml | 9 +- .../examples/sidecar-adapt-in.py | 19 +-- .../examples/sidecar-adapt-out.py | 1 + .../tracegrad-harness/export-prompt/SKILL.md | 107 ++---------- .../tracegrad-harness/import-traces/SKILL.md | 111 ++----------- .../import-traces/jsonl-contract.md | 28 ++++ skills/tracegrad-harness/next-batch/SKILL.md | 142 ++-------------- .../tracegrad-harness/propose-edits/SKILL.md | 117 +++---------- .../tracegrad-harness/review-edits/SKILL.md | 157 ++++-------------- 10 files changed, 163 insertions(+), 616 deletions(-) create mode 100644 skills/tracegrad-harness/import-traces/jsonl-contract.md diff --git a/skills/tracegrad-harness/README.md b/skills/tracegrad-harness/README.md index 66f1533..76b189f 100644 --- a/skills/tracegrad-harness/README.md +++ b/skills/tracegrad-harness/README.md @@ -1,79 +1,35 @@ # Tracegrad harness skill pack -A verb-first skill pack so Claude Code or Pi can drive Tracegrad from -**outside** the Python package. Core stays lean and harness-driven. This -directory is documentation and examples, not a product surface and not a -dependency. +Harness loop for Claude Code / Pi. Core stays lean. Sidecars live beside the user repo, never under `src/tracegrad/`. No Kitaru in this pack. -Kitaru is not in core. Do not add Kitaru code, deps, or docs from this pack. -Adapters live beside the user's repo (sidecar), never under `src/tracegrad/`. +`tracegrad run` does not write the prompt. Harness apply is only `tracegrad apply --accept ` after a human or policy names those indices — procedure in [review-edits](review-edits/SKILL.md). Never invent accepts. -## The loop +CLI flags: `tracegrad --help`. Manifest, `.tracegradrc`, and project state: repo README. -1. **Import traces.** A sidecar adapt-in maps the user's existing store into - Tracegrad JSONL. Their pipeline does not change. Tracegrad never learns the - stack. -2. **Propose edits.** `tracegrad run --estimate`, then `tracegrad run`. Analysis - writes cards and a proposal under `.tracegrad/`. It does **not** write the - prompt. -3. **Review edits.** Default: stop and ask the human. **Unattended** apply - needs a local policy file that is present **and** clearly permits it. - **Attended** apply is allowed after the human explicitly names card - indices (`tracegrad apply --accept …`). Missing or ambiguous policy does - not block that attended path; it only forbids applying with no - human-named list. Never invent `--accept` indices. -4. **Export prompt.** After a successful `tracegrad apply`, a sidecar adapt-out - copies the written template back to the path the user's app actually loads - (no-op if that path already *is* the manifest `template_file`). -5. **Next batch.** Thin conductor: import → estimate → propose → review → - export → `tracegrad status` / `tracegrad trends`. Unattended apply stays - off unless the policy file both exists and permits it. Attended apply - after explicit human indices is allowed; otherwise stop at review. +## Loop -The existing gate is unchanged: `tracegrad run` never writes the prompt; only -`tracegrad apply` does, and only after a human or an explicit policy accept. +1. [import-traces](import-traces/SKILL.md) — adapt-in a user-named export → JSONL +2. [propose-edits](propose-edits/SKILL.md) — estimate, then run (or attribute + propose); cards on disk +3. [review-edits](review-edits/SKILL.md) — show cards; `--accept` after human- or policy-named indices +4. [export-prompt](export-prompt/SKILL.md) — adapt-out after apply wrote the template +5. [next-batch](next-batch/SKILL.md) — conductor over 1–4, then `status` / `trends` -## Skills +## Reach -| Skill | Folder | Does | -| --- | --- | --- | -| import traces | [`import-traces/`](import-traces/SKILL.md) | Adapt-in → JSONL | -| propose edits | [`propose-edits/`](propose-edits/SKILL.md) | Estimate + run (or attribute + propose); stop with cards | -| review edits | [`review-edits/`](review-edits/SKILL.md) | Show cards; `--accept` only after human or policy names indices | -| export prompt | [`export-prompt/`](export-prompt/SKILL.md) | Adapt-out after apply | -| next batch | [`next-batch/`](next-batch/SKILL.md) | Conduct the loop; unattended apply needs policy, else stop at review | +| Situation | Skill | +| --- | --- | +| import traces, adapt-in, JSONL from a store | [import-traces](import-traces/SKILL.md) | +| estimate, propose, run, attribute | [propose-edits](propose-edits/SKILL.md) | +| cards, review, accept, apply, policy | [review-edits](review-edits/SKILL.md) | +| export / adapt-out the applied prompt | [export-prompt](export-prompt/SKILL.md) | +| next batch, close the loop | [next-batch](next-batch/SKILL.md) | -## Policy file +JSONL ingest rules: [import-traces/jsonl-contract.md](import-traces/jsonl-contract.md). -The harness agent may read a project-local policy file (suggested name: -`tracegrad-apply-policy.toml` at the project root, or a path the user names). -**Tracegrad core does not load this file.** `.tracegradrc` remains the only -config core reads (`neverDelete`, coverage, harness presets). The policy file -is an extra, agent-side gate on whether the skill may invoke `tracegrad apply`. +## Sidecars and policy -Shape (see [`examples/policy.commented.toml`](examples/policy.commented.toml)): +Copy [examples/](examples/) beside the user repo: -- `unattended_apply` — default **false**. Off means stop and ask. -- `accept` — list of integers (TOML array, e.g. `accept = [0, 2]`). The CLI - flag is still comma-separated: `tracegrad apply --accept 0,2`. Empty, - omitted, or guessed → do not apply. -- `allow_delete` — default **false**. Skip or refuse `DELETE` edits. -- `neverDelete` — instruction ids the agent must not apply, even if they - survived core gates. Complements `.tracegradrc`; does not replace it. -- `token_ceiling` — if the proposal's `tokens_after` would exceed this, stop - and ask rather than apply. - -If the file is missing, `unattended_apply` is false, `accept` is empty, or any -rule is ambiguous: **do not apply unattended** — stop and ask. A human may -still name indices for attended `--accept`. Never pass `--all`; always pass -the explicit `--accept` list. - -## Adapters stay outside core - -Copy the stubs in [`examples/`](examples/) next to the user's repo: - -- `sidecar-adapt-in.py` — foreign traces → JSONL contract +- `sidecar-adapt-in.py` — `FIELD_MAP` = Tracegrad field → foreign path - `sidecar-adapt-out.py` — applied template → user path - -Do not vendor these into `src/tracegrad/`. Do not add a Kitaru (or any other -eval-stack) integration to the package to make import/export "just work". +- `policy.commented.toml` — agent-side apply gate (unattended apply off by default; `accept` is a TOML integer array). Core does not load this file. diff --git a/skills/tracegrad-harness/examples/policy.commented.toml b/skills/tracegrad-harness/examples/policy.commented.toml index f16ce8e..bcfa18c 100644 --- a/skills/tracegrad-harness/examples/policy.commented.toml +++ b/skills/tracegrad-harness/examples/policy.commented.toml @@ -1,13 +1,14 @@ # Tracegrad harness apply policy — EXAMPLE # -# Read by Claude Code / Pi skill pack (review-edits, next-batch). +# Agent-side gate for review-edits (next-batch follows that skill). # NOT loaded by Tracegrad core. Core still only writes the prompt via -# `tracegrad apply`, and only for the indices passed to `--accept`. +# `tracegrad apply --accept `. # # Copy to the project (suggested name: tracegrad-apply-policy.toml) and # edit. Unattended apply stays OFF until a human sets unattended_apply -# and names accept indices. Missing file, false, or ambiguous → the -# skill stops and asks. Never invent --accept values. +# and names accept indices. Missing file, false, or ambiguous → stop +# and ask. A human may still name indices for attended --accept. +# Never invent --accept values. There is no allow_all. # Fully unattended apply is OFF by default. Leave this false (or omit it). unattended_apply = false diff --git a/skills/tracegrad-harness/examples/sidecar-adapt-in.py b/skills/tracegrad-harness/examples/sidecar-adapt-in.py index 5588dc8..57c8bb5 100644 --- a/skills/tracegrad-harness/examples/sidecar-adapt-in.py +++ b/skills/tracegrad-harness/examples/sidecar-adapt-in.py @@ -3,21 +3,14 @@ Copy this file next to the user repo. Do not move it into src/tracegrad/. The user pipeline stays unchanged; this adapter is the only place that -learns the foreign field names. +learns the foreign field names. This stub is not a vendor integration. -JSONL contract (one object per line, extra keys forbidden by ingest): +One JSON object per line; ingest forbids extra keys. Required: trace_id, +input, output, judge.score in [0, 1], judge.rationale, prompt_hash. +Optional meta.model. Ingest drops duplicate trace_ids, rationales under +24 usable characters, and non-dominant prompt_hash. - { - "trace_id": str, - "input": str, - "output": str, - "judge": {"score": float in [0, 1], "rationale": str}, - "prompt_hash": str, - "meta": {"model": str} # optional - } - -Fill FIELD_MAP (or --map-json) for the local export. This stub is not a -vendor integration. +FIELD_MAP (or --map-json) is Tracegrad field → foreign dotted path. """ from __future__ import annotations diff --git a/skills/tracegrad-harness/examples/sidecar-adapt-out.py b/skills/tracegrad-harness/examples/sidecar-adapt-out.py index 6d0306a..63dd81c 100644 --- a/skills/tracegrad-harness/examples/sidecar-adapt-out.py +++ b/skills/tracegrad-harness/examples/sidecar-adapt-out.py @@ -3,6 +3,7 @@ Run only after `tracegrad apply` has written the manifest template. Copy this file next to the user repo. Do not move it into src/tracegrad/. +This adapter does not apply edits and does not invent a destination. If --from and --to are the same path, this is a no-op. That is the usual case when the app already loads the manifest `template_file`. diff --git a/skills/tracegrad-harness/export-prompt/SKILL.md b/skills/tracegrad-harness/export-prompt/SKILL.md index cca07a6..db32302 100644 --- a/skills/tracegrad-harness/export-prompt/SKILL.md +++ b/skills/tracegrad-harness/export-prompt/SKILL.md @@ -1,109 +1,36 @@ --- name: export-prompt -description: After a successful tracegrad apply, copy the written template back to the user path via a sidecar adapt-out. Use when asked to export, sync, or deploy the applied prompt. Does not apply. +description: "Adapt-out the applied template to a user-named path. Invoke for export after apply. Not apply." --- # Export prompt -Sidecar adapt-out: after `tracegrad apply` has already written the template, -copy or write that file to the path the user's app actually loads. +Sidecar adapt-out: copy the template `tracegrad apply` already wrote to the path the user's app loads. Apply is [`../review-edits/SKILL.md`](../review-edits/SKILL.md). Export is not a Tracegrad subcommand. -This skill does **not** apply edits. If apply has not happened, stop and send -the user to `review-edits`. - -## When to use - -- Apply succeeded and the production prompt is a *different* path than the - manifest `template_file`. -- The user asks to export, sync, or copy the applied prompt back. -- `next-batch` reaches the export step after a permitted apply. - -Skip (successful no-op) when the manifest path **is** the user path. Do not -use this skill to bypass the apply gate by writing a "proposed" prompt. - -## Sidecar adapt-out - -Tracegrad writes the file named in the manifest, resolved against -`--base-directory`. The adapter lives beside the user repo. Copy -[`../examples/sidecar-adapt-out.py`](../examples/sidecar-adapt-out.py) and -point `--from` at the applied template and `--to` at the user path. - -```sh -python sidecar-adapt-out.py \ - --from path/from/manifest/prompt.md \ - --to /path/the/user/app/loads/prompt.md -``` - -Do not add an export command to Tracegrad core. Do not vendor the user's -prompt store into `src/tracegrad/`. +Skip (successful no-op) when the manifest `template_file` **is** the user path. Do not copy a proposed, unapplied template. ## Steps -1. Confirm apply already happened for this run: new prompt hash on stdout, or - an entry in `.tracegrad/ledgers/applied.jsonl`, plus a snapshot under - `.tracegrad/snapshots/`. If none, **stop** — export has nothing safe to - copy. - -2. Resolve the source path: manifest `template_file` + `--base-directory`. - -3. Resolve the destination: only a path the user named (config, flag, or - existing sidecar defaults). Do not guess a production path. - -4. Run the sidecar. Overwrite only that destination. - -5. Report source, destination, and that core was not modified. - -## Inputs / outputs - -**Inputs** - -- Applied template path (manifest `template_file`). -- User destination path. -- Sidecar script (example: `examples/sidecar-adapt-out.py`). -- Optional: `--project-root` / `--base-directory` used during apply, so the - same file is found. - -**Outputs** - -- User-path file updated to match the applied template (or no-op if identical - path / identical bytes). -- No Tracegrad state writes. No second apply. +1. **Confirm apply for this run.** Look for a new prompt hash on apply stdout, a new line in `.tracegrad/ledgers/applied.jsonl`, and a snapshot under `.tracegrad/snapshots/`. -## Failure modes + Done: at least one of those exists. If none, stop and send the user to `review-edits`. -| Failure | What to do | -| --- | --- | -| No apply yet | Stop. Do not copy a pre-apply template and call it exported. | -| Stale proposal was refused | Nothing to export. Re-run propose + review. | -| Destination unknown | Stop and ask. | -| Destination outside what the user named | Refuse. | -| Apply succeeded but template hash does not match apply output | Stop; do not overwrite the user path with a file that may have been edited out of band. | -| Urge to `tracegrad apply` "so there is something to export" | Refuse unless `review-edits` policy/human already allowed it. | +2. **Source.** Resolve manifest `template_file` against the same `--base-directory` used at apply (`tracegrad apply --help`). The source is the path apply printed (`applied … to