Production → test: side-effect capture, behavioral fingerprints, and run conversion

[AminChirazi]

Production → test: convert real agent runs into committed regression flows

Context — what already exists (do not rebuild)

The original draft of this issue specified five subsystems. Four of them are the
shipped product; this rewrite scopes the issue down to the genuine delta.

Capability	Where it lives today
Execution engine (record / replay)	flowproof-agent (record), flowproof-replay (zero-LLM replay)
Trace capture (LLM, tools, MCP, HTTP mocks)	trace format v1, cassette + MCP lanes (docs/trace-format.md)
Event storage / timeline	JSON-lines traces, schema-enforced (crates/flowproof-trace/schema/)
Assertions / evaluation	assert_tool_call, assert_no_tool_call, assert_no_egress, controls, flowproof audit --since
Replay mismatch detection	strict positional matching + field-level diff (the product's core)
CI execution	flowproof run in CI, no API key, no model calls
Secret handling	invariant 9 + secret-scan corpus

Vocabulary rule: this work uses the repo's existing terms — flow, spec,
trace, cassette, control, heal — and extends the existing trace format. No
parallel event model, no new storage layer, no TypeScript (the engine is a
single Rust binary; CHARTER §3).

What this issue actually adds

Every production failure becomes a reproducible test.

Today a trace is born from a deliberate flowproof record. The delta is
making any real run — including a production failure — convertible into a
committed flow + trace pair that replays forever.

1. Side-effect capture (extend, don't invent)

Filesystem observation already reports unlink/rename/truncate on Linux.
Extend the same observation approach to a first-class, asserted side-effect
record in the trace:

side_effect: { kind: fs_write | http_request | db_change | sap_transaction,
               target, before?, after?, diff? }

• Phase A covers fs_* and http_request (both observable at existing
  boundaries). db_change and sap_transaction ride on their adapters later.
• Trace-format change ⇒ docs/trace-format.md + JSON Schema in the same
  commit (invariant 5), and needs-human review (CHARTER §8).

2. flowproof convert — production run → regression flow

New verb alongside record / run / heal:

flowproof convert <captured-run> --out flows/regressions/<name>

Pipeline: filter noise → extract model/tool interactions into a cassette →
extract side effects → suggest assertions (never auto-commit; a cassette
entering the repo is a human act, invariant 8) → emit flow YAML + trace as a
reviewable diff.

Like healing: a proposal, never a silent mutation (invariant 4).

3. Behavioral fingerprint

Derived (not stored authority) summary of a trace:

fingerprint: { tools[], tool_sequences[], endpoints[], models[],
               side_effect_kinds[], graph_hash, outcome_class }

Used for: dedup of converted runs, drift detection between recordings, and
later as Guard's baseline (#466). Lives in flowproof-trace as a pure
function of a trace; deterministic, no model calls.

4. Multi-model comparison (last, optional)

flowproof record already takes a backend; add a comparison runner that
records the same spec against N backends and diffs the fingerprints/verdicts.
Explicitly deferred until 1–3 are green — it is the only part of this
issue that spends model tokens.

Non-goals

• No new execution engine, storage layer, or event model — extend trace v1.
• No HYBRID replay mode until a concrete need is observed (frequency gate).
• No DB adapter in this issue — a new app: target is a product decision
  (CHARTER §3); db_change side effects arrive via existing adapters only.
• No production sampling infrastructure — conversion is pull, not push.
• No semantic/LLM evaluators here — deterministic assertions only.

Constraints (from CHARTER.md — binding)

1. Replay stays zero-LLM, zero-API-key (invariants 1–2).
2. Trace-format + schema + docs move in the same commit (invariant 5).
3. Converted cassettes are committed only by a human (invariant 8).
4. No secret ever reaches a trace (invariant 9) — convert runs the secret
   scan before emitting anything.
5. PRs stay under ~400 lines; this issue is a sequence of small PRs.
6. This work is scheduled after Milestone 2 (Agent-boundary testing follow-ups: Anthropic API, streaming replay, MCP-server mocking #61 + falsifiability) unless
   the charter is amended to say otherwise. Recording it now, building it
   in order.

Plan

Phase A — side effects in the trace (format + capture)

• side_effect record in trace v1 (+ schema + docs, same commit)
• fs side-effect capture promoted from report to recorded effect (Linux)
• http_request side effects from the existing mock/containment boundary
• falsifiability: a red-path fixture proving a side-effect assertion can fail

Phase B — flowproof convert

• converter: captured run → flow YAML + cassette (reviewable diff)
• assertion suggester (deterministic: tools called / not called / effects)
• secret scan gate in the convert path
• committed example under examples/ proving a converted run replays green

Phase C — fingerprints

• fingerprint(trace) in flowproof-trace + flowproof fingerprint CLI
• fingerprint diff between two traces (drift report)

Phase D — multi-model comparison (optional, token-spending)

• comparison runner + fingerprint/verdict diff across backends

Definition of done

A real (or simulated-production) agent run is converted by flowproof convert
into a flow + trace that: replays green with zero LLM calls and no API key,
carries at least one side-effect assertion, fails when the recorded behavior is
perturbed (non-vacuity, R4), and entered the repo through a human-reviewed diff.

[AminChirazi]

Phase A implemented as a five-PR stack (needs-human, per CHARTER §8 — trace-format change):

1. feat(trace): first-class side_effect record and lane (#465 Phase A, 1/5) #468 feat(trace): the side_effects lane — serde types + side-effect-v1.schema.json + docs/trace-format.md in one range (squash-merge, invariant 5) + the committed fixture
2. feat(agent): record observed fs writes into the side-effect lane (#465 Phase A, 2/5) #469 feat(agent): fs capture promoted from stderr report to recorded effect — total path-hygiene function (workspace-relative or hash-redacted; traversal forms always redact), lane inside the secret-scan store-guard
3. feat(agent): record observed egress as http_request side effects (#465 Phase A, 3/5) #470 feat(agent): http_request side effects from the containment boundary — EgressLog.observed, ${VAR} targets stored unresolved, Containment::observation_only() so observation never claims enforced
4. feat(trace): assert_no_side_effect turns side-effect observation into a verdict (#465 Phase A, 4/5) #471 feat(trace): assert_no_side_effect — cannot certify honesty off-Linux, sentinel-prefix verdict classification (agent-chosen filenames cannot relabel a violation)
5. test(falsifiability): the side-effect assertion proven able to fail (#465 Phase A, 5/5) #472 test(falsifiability): the red-path proof — fixture violating the property, real assertion path, verdict FAIL, non-zero exit; verified non-vacuous (gutted verdict ⇒ harness fails)

The design's plan was 4 PRs; 4 was split into 4+5 because the falsifiability harness alone is ~290 lines and the ratchet caps a PR range at 400. All five ranges pass scripts/gate/ratchets.sh (400/400/396/395/370) and the full local gate incl. cargo check --target x86_64-unknown-linux-gnu. flowproof run scripts/demo/order-status.flow.yaml replays green with no credentials in the environment.

Deferred as needs-human (deliberately not guessed): a JSON Schema for the whole agent-trace document; recording the supervisor's structural refusals (SCM_RIGHTS/listen) as trace data; a side-effect summary on the run record's Evidence; hostname enrichment for http_request targets. One dependency-policy question is flagged in #469 (direct sha2 dep).