Implementation plan: tracegrad v0.1.0 (spec v0.2)

[dnth]

Implementation plan for tracegrad v0.1.0, per design spec v0.2 (post-review). Python >= 3.11, managed with uv. Runtime deps: httpx, pydantic v2. Dev: pytest.

Design invariants (non-negotiable)

• Determinism boundary: exactly two modules import llm — attribute and synthesize. Enforced by a lint test, not convention.
• Analysis never writes the prompt. apply is the only writing command.
• Evidence is mechanical: every quote substring-verified in code against the persisted distilled store, with a declared source field. Violations require an output quote.
• Cross-batch trends are advisory — CIs and detectable-effect floors displayed, no automatic reverts.
• Zero proposed edits is a valid, expected output.

Build order

Stage 1 — Contracts and state

• schema.py: Manifest (template file, engine, vars, sampling, judge fingerprint), Trace (prompt_hash required), AttributionResult (theme_slug on violations/harmful, quote_source), Edit (instruction_id-addressed, covers_theme, watch_metric), Report (applied_prompt_hash, cluster {theme, numerator, denominator}), StepVerdict
• config.py: rc loading, neverDelete patterns, minEffect, minCoverage, convergenceRuns, harness presets
• state.py: .tracegrad/ layout, O_EXCL lock, atomic writes (tmp + fsync + os.replace), append-only JSONL ledgers, per-run resume state, .gitignore written by init

Stage 2 — Deterministic core (no model, golden fixtures)

• ingest.py: validation, prompt-hash partitioning (dominant partition only), judge fingerprint check, judge canary comparison, meta.model partitioning, rationale-quality floor
• distill.py: template render (declared engine), reduction, redaction with stable placeholders, content-addressed distilled/ store, distill_config_hash
• inventory.py: versioned segmenter (bullets/numbered items/sentences, abbreviation-safe), content fingerprint + stable lineage_id, duplicate-text ordinals, origin: variable spans marked non-editable
• aggregate.py: instruction table keyed by lineage, unified theme space (exact slug then single-link Jaccard >= 0.6 in sorted order), rates as {numerator, denominator} over the whole batch, gap ledger (graduation needs >= 2 distinct runs or sessions; retirement only on improved trend or explicit human resolve; restore on revert)

Stage 3 — Edit engine and gates (pure, tested before any model)

• edits.py: normalization N() (NFKC, whitespace collapse, quote/dash folding) with offset map, span resolution by instruction_id + checksum, overlap rejection, right-to-left application, ADD anchor semantics (sibling after full span, indentation inherited, START/append sentinels), orphan-anchor rejection
• gates.py: G1 edit cap; G2 accounting by re-diff; G3 REWRITE->ADD reclassification via clause-level diff (new imperative clause without token-overlap counterpart); G4 source-checked quotes + negation-window flag; G5 template-measured budget, zero-sum at ceiling; G6 rejection/revert memory with distinct-session bar; G7 variable spans untouchable; G8 duplicate-ADD rejection. Per-edit drop with named reason — never run-level abort

Stage 4 — Model layer and attribution

• llm.py: backend registry with complete(system, user, *, cacheable_prefix=None, schema=None). openai: params only when set, capability-flagged reasoning/effort with retry-without fallback, OpenRouter HTTP-200 error body check, Retry-After honored, 900s synthesis timeout without retry-on-timeout, shared client, jobs-way concurrency. claude: isolated preset (--allowed-tools "", --strict-mcp-config --mcp-config '{}', empty --settings, system prompt passed properly, is_error checked). Generic harness presets + inline command. Defaults: attribution -> openai, synthesis -> claude, per-tier availability resolution
• attribute.py: per-trace call at temperature 0 where supported, canonical theme vocabulary fed forward, full instrument-versioned cache key, minCoverage abort, blinded-attribution health sample with reported agreement rate

Stage 5 — Synthesis, trends, apply, CLI

• synthesize.py: thin — prompt assembly, per-edit re-prompt loop (max 2), "no edits warranted" path, failed-proposal autopsy dump
• trends.py: two-proportion z-tests, CI on the difference, verdicts improved/regressed/eliminated/no-signal with minEffect, same-instrument re-attribution of affected clusters, all-cluster guardrails, two-consecutive-regressed hysteresis, convergence state, detectable-effect floors for status
• apply.py: review cards (diff, evidence + sources, flags), write-back of applied_prompt_hash + accepted list, base-hash re-check, pre-write snapshot + revert, out-of-band-edit staleness marking
• cli.py: run (with --estimate), attribute, propose, trends, apply, status, init

Stage 6 — Tests and release

• Unit: segmenter golden fixtures, N() and span ordering, every gate individually, ledger graduation/retirement/restore across simulated runs, trend math vs hand-computed values, denominator correctness
• E2E on FakeBackend: fixture batch -> report -> partial acceptance -> resume-after-kill -> byte-exact applied prompt
• Opt-in harness smoke test: one isolated claude -p round-trip pinning JSON shape and is_error
• example/ with a synthetic batch + manifest; README quickstart verified against it
• Publish to PyPI

Explicitly deferred (post-v0.1.0)

• A/B mode (spec section 11.1) — the primary verification upgrade; per-trace prompt_hash already carries arm assignment, so no schema change needed
• Replay hook (spec section 11.2) with ingest-time holdout reservation
• jinja-basic template engine (v0.1.0 ships none + format)

Estimate

One to two weeks of careful work. The gates, edit engine, trend statistics, and state hardening are the modules that punish haste — they are also the product.

[dnth]

Stages 2-6 shipped in #3 (squashed to 3a5e008). Stage 1 shipped in #2.

Publishing to PyPI is dropped, so this issue is complete as far as it is going to
be. The tag-triggered release workflow stays in the repo, dormant — it only fires on a
v* tag, and nothing tags automatically. Cutting v0.1.0 later needs a pypi
environment with trusted publishing; nothing else.

What landed

All of Stages 2-5 and the testing half of Stage 6:

• Deterministic core — ingest, distill, inventory, aggregate, plus canonical
  and normalize. Rates over the whole batch, mechanical theme unification, the gap ledger.
• Edit engine and gates — edits and gates, G1-G8, per-edit drops with named reasons.
• Model layer — llm, attribute, synthesize. Attribution measures at temperature 0
  with a versioned instrument fingerprint; the isolated claude preset runs with no tools,
  no MCP, no inherited settings.
• Trends, apply, CLI — advisory statistics with CIs and detectable-effect floors,
  the human gate as the only writer, and run/attribute/propose/trends/apply/
  status/init.
• 331 tests, ruff pinned and running in CI, and a worked example/ the README
  quickstart is verified against.

Departures from the plan, both deliberate

• pipeline.py is a module the plan did not list — orchestration lives there so cli.py
  stays thin and a run is testable as a function.
• ports.py holds the Backend protocol so the orchestrator can pass a backend around
  without importing the model layer. The determinism boundary is enforced by AST in
  tests/test_boundary.py, including the laundering case.

Carried forward

Same-instrument re-attribution is half-built on purpose. Reports carry an instrument
fingerprint and a cross-instrument comparison is refused rather than silently differenced,
but the previous batch is not re-attributed when the instrument changes — that re-spends
the attribution budget and should be an explicit choice. Worth its own issue if it is ever
wanted.

Deferred items are untouched as specified: A/B mode, the replay hook, and jinja-basic.
TemplateEngine is exactly {none, format} and rendering raises on anything else.