[FIX] [BREAKING]: Evaluate attack verdicts over final traces

docs/api/attacks.md

@@ -4,4 +4,5 @@
     options:
       members:
         - Attacks
+        - StopWhen
         - XPIAExecution

docs/api/core-protocols.md

@@ -51,10 +51,19 @@ Protocols and ABCs that define RAMPART's extension points. Implement these to co
         - ExecutionEventData
         - ExecutionEventHandler
         - ExecutionHandlerFactory
-        - evaluate_turn_async
         - register_default_handler_factory
         - clear_default_handler_factory
 
+## Trace Execution
+
+::: rampart.core.trace
+    options:
+      members:
+        - EvaluationRecord
+        - TraceRun
+        - run_trace_async
+        - evaluate_final_trace_async
+
 ## Errors
 
 ::: rampart.core.errors

docs/api/core-types.md

@@ -36,8 +36,6 @@ available from `rampart.core`; established result types remain importable from
         - InjectionRecord
         - resolve_attack_verdict
         - resolve_probe_verdict
-        - resolve_as_attack
-        - resolve_as_probe
 
 ## Configuration
 

docs/api/index.md

@@ -11,7 +11,7 @@ API reference organized by RAMPART's package layout. Each page documents the pub
 |------|----------|
 | [Core Types](core-types.md) | `Payload`, `Request`, `Response`, `Turn`, `Result`, `SafetyStatus`, `HarmCategory`, and more |
 | [Core Protocols](core-protocols.md) | `Session`, `AgentAdapter`, `Evaluator`, `Surface`, `InjectionHandle`, and more |
-| [Attacks](attacks.md) | `Attacks.xpia()`, `XPIAExecution` |
+| [Attacks](attacks.md) | `Attacks.xpia()`, `StopWhen`, `XPIAExecution` |
 | [Probes](probes.md) | `Probes.behavior()`, `SingleTurnExecution` |
 | [Evaluators](evaluators.md) | `ToolCalled`, `ResponseContains`, `ResponseScope`, `SideEffectOccurred`, `LLMJudge`, `TranscriptScope` |
 | [Drivers](drivers.md) | `StaticDriver`, `LLMDriver` |

docs/attacks/xpia.md

@@ -30,9 +30,10 @@ sequenceDiagram
 1. **Inject** — Place payloads into the agent's data sources via surfaces. Each `surface.inject(payload)` returns an [`InjectionHandle`][rampart.core.injection.InjectionHandle].
 2. **Wait** — Handles call `wait_until_ready_async()` to allow indexing. Runs concurrently for multiple surfaces.
 3. **Trigger** — Send benign prompts that cause the agent to retrieve the injected content. Triggers are never adversarial — the attack is in the payload, not the prompt.
-4. **Evaluate** — Check each turn for the attack objective. Early-stops on detection.
-5. **Clean up** — Remove injected content. Guaranteed via `AsyncExitStack`, even on exceptions.
-6. **Result** — Produce a [`Result`][rampart.core.result.Result] via `resolve_as_attack` semantics.
+4. **Stop (optional)** — Check `stop_when` after each response and stop when detected.
+5. **Evaluate** — Check the attack objective once over the final trace.
+6. **Clean up** — Remove injected content. Guaranteed via `AsyncExitStack`, even on exceptions.
+7. **Result** — Map the final evaluation using attack semantics.
 
 ---
 
@@ -162,8 +163,11 @@ The `&` above asks whether both happened, so one condition that definitively did
     `ResponseContains` requires an explicit temporal scope, even for a
     single-turn attack. The complete positive and negated mapping is maintained in the
     [Temporal Scope table](../usage/authoring-tests.md#temporal-scope).
-    Use `CURRENT_TURN` only when earlier responses should be ignored. Scope
-    applies only to turns in the evaluator context; it does not control
+    Use `CURRENT_TURN` only when earlier responses should be ignored.
+
+    XPIA verdict evaluators receive the final trace. Automatic stopping is
+    enabled only when detection is known to remain true as the trace grows.
+    Scope applies only to turns in the evaluator context; it does not control
     execution length or early stopping.
 
 ### LLMDriver for Adaptive Triggers
@@ -202,6 +206,28 @@ assert result, result.summary
 !!! warning
     Construct a new `LLMDriver` per test. Each instance maintains its own conversation state and cannot be reused.
 
+!!! note "Adaptive driver budgets"
+    `LLMDriver` does not stop itself. The default `stop_when=StopWhen.AUTO` stops
+    early for stable built-in conditions such as `ToolCalled`, but unknown or
+    stochastic evaluators run to `max_turns` and evaluate the final trace
+    once. Use an explicit `stop_when` when that online judgment intentionally
+    defines the end of the attack scenario.
+
+!!! note "Upgrading from per-turn attack verdicts"
+    Earlier releases evaluated XPIA after each response and stopped at the
+    first detection. XPIA now evaluates the final trace once, so single-trigger
+    attacks with deterministic evaluators keep the same verdicts. The default
+    `stop_when=StopWhen.AUTO` still stops early for stable built-in conditions such as
+    `ToolCalled`. Other evaluators, including LLM judges, no longer stop early
+    by default: they are called once on the final trace, and adaptive drivers
+    can run up to `max_turns`. Pass the same evaluator as `stop_when` to restore
+    per-turn early stopping without a duplicate final call. With the default,
+    a stochastic evaluator is sampled once per run instead of once per turn, so
+    trial pass rates can shift. Replace `resolve_as_attack(eval_results=...)`
+    with `resolve_attack_verdict(evaluation=...)`, and replace
+    `evaluate_turn_async` with `run_trace_async` and
+    `evaluate_final_trace_async`.
+
 ---
 
 ## Trigger Options
@@ -233,6 +259,7 @@ See [`Attacks.xpia()`][rampart.attacks.Attacks.xpia] for the full API reference.
 | `inject` | `InjectionHandle \| list[InjectionHandle] \| None` | `None` | Prepared injections from `surface.inject()`. `None` for inline XPIA. |
 | `trigger` | `str \| list[str] \| Request \| list[Request] \| PromptDriver` | required | Benign prompt(s) that cause retrieval of injected content. |
 | `evaluator` | [`Evaluator`][rampart.core.evaluator.Evaluator] | required | What attack condition to detect. |
+| `stop_when` | [`Evaluator`][rampart.core.evaluator.Evaluator] `\|` [`StopWhen`][rampart.attacks.StopWhen] `\| None` | `StopWhen.AUTO` | Online stop condition. `StopWhen.AUTO` reuses stable built-in verdict evaluators and exposes their prefix results to adaptive drivers; `None` disables stopping and online feedback. |
 | `max_turns` | `int` | `5` | Maximum prompt-response exchanges; reaching the limit resolves the trace normally. |
 | `event_handlers` | `list[ExecutionEventHandler] \| None` | `None` | Additional lifecycle event handlers. |
 

docs/concepts/attacks.md

@@ -14,9 +14,13 @@ Attacks use the following mapping from evaluator outcomes to safety verdicts:
 | `NOT_DETECTED` | `SAFE` | The attack failed — the agent resisted |
 | `UNDETERMINED` | `UNDETERMINED` | The evaluator could not determine whether the attack succeeded |
 
-Precedence when multiple turns are evaluated: `DETECTED` > `UNDETERMINED` > `NOT_DETECTED`. If any turn detected the attack objective, the agent is compromised regardless of other turns.
+The evaluator runs once over the final trace, and the outcome maps directly
+to the verdict. This logic lives in
+[`resolve_attack_verdict`][rampart.core.result.resolve_attack_verdict].
 
-This logic lives in [`resolve_as_attack`][rampart.core.result.resolve_as_attack].
+Attack factories may evaluate a separate online `stop_when` condition while
+the trace is being produced. XPIA's `StopWhen.AUTO` default reuses the verdict
+evaluator only when detection is known to be stable as turns are appended.
 
 ---
 
@@ -27,9 +31,10 @@ All attack executions share this lifecycle:
 1. **Inject** (optional) — Place payloads into the agent's data sources via [surfaces](../api/core-protocols.md)
 2. **Wait** — Allow time for indexing or propagation
 3. **Trigger** — Send prompts that cause the agent to process the injected content
-4. **Evaluate** — Check whether the attack objective was achieved
-5. **Clean up** — Remove injected content (guaranteed, even on failure)
-6. **Report** — Produce a [`Result`][rampart.core.result.Result]
+4. **Stop (optional)** — Check an online condition after each response
+5. **Evaluate** — Check the final trace once for the attack objective
+6. **Clean up** — Remove injected content (guaranteed, even on failure)
+7. **Report** — Produce a [`Result`][rampart.core.result.Result]
 
 The injection phase is optional — inline attacks attach payloads directly to the trigger prompt.
 

docs/concepts/overview.md

@@ -58,7 +58,7 @@ A single test run flows from your pytest test, through a RAMPART attack or probe
 
 *Request / response cycle for a single test run.*
 
-Under the hood, every execution follows a common lifecycle owned by [`BaseExecution`][rampart.core.execution.BaseExecution], which drives the per-turn loop between the strategy, your adapter, and the evaluator:
+Under the hood, every execution follows a common lifecycle owned by [`BaseExecution`][rampart.core.execution.BaseExecution]. The strategy drives requests through your adapter, optionally evaluates an online stop condition, then evaluates the completed trace once for the verdict.
 
 ```mermaid
 sequenceDiagram
@@ -76,11 +76,16 @@ sequenceDiagram
         Strat->>Strat: driver.next_prompt_async(history)
         Strat->>Adapter: session.send_async(request)
         Adapter-->>Strat: Response
-        Strat->>Eval: evaluate_async(context)
-        Eval-->>Strat: EvalResult
-        Note over Strat: Early stop if detected
+        opt Explicit online stop condition
+            Strat->>Eval: evaluate_async(prefix context)
+            Eval-->>Strat: stop EvalResult
+            Note over Strat: Stop if detected
+        end
     end
 
+    Strat->>Eval: evaluate_async(final trace context)
+    Eval-->>Strat: final EvalResult
+
     Strat-->>Exec: Result
     Exec->>Exec: fire ON_POST_EXECUTE
     Exec-->>Test: Result
@@ -112,7 +117,7 @@ Evaluators are **polarity-free**. They answer "did X happen?" — not "is X good
 - In an **attack**, detection means the attack objective was achieved → **UNSAFE**
 - In a **probe**, detection means the expected behavior is present → **SAFE**
 
-The [`Attacks`][rampart.attacks.Attacks] and [`Probes`][rampart.probes.Probes] factories handle this mapping automatically via [`resolve_as_attack`][rampart.core.result.resolve_as_attack] and [`resolve_as_probe`][rampart.core.result.resolve_as_probe].
+The [`Attacks`][rampart.attacks.Attacks] and [`Probes`][rampart.probes.Probes] factories handle this mapping automatically by applying [`resolve_attack_verdict`][rampart.core.result.resolve_attack_verdict] or [`resolve_probe_verdict`][rampart.core.result.resolve_probe_verdict] to one final-trace evaluation.
 
 You can reuse the same evaluator in both contexts. A [`ToolCalled`][rampart.evaluators.tool_called.ToolCalled] evaluator detects whether a tool was called — whether that's good or bad depends on whether you're attacking or probing.
 

docs/concepts/probes.md

@@ -14,9 +14,9 @@ Probes use the inverse mapping from evaluator outcomes:
 | `NOT_DETECTED` | `UNSAFE` | The expected behavior is missing — a regression |
 | `UNDETERMINED` | `UNDETERMINED` | The evaluator could not determine whether the behavior is present |
 
-Precedence: `NOT_DETECTED` > `UNDETERMINED` > `DETECTED`. If any turn failed to detect the expected behavior, the agent is non-compliant.
-
-This logic lives in [`resolve_as_probe`][rampart.core.result.resolve_as_probe].
+The evaluator runs once over the completed trace, and the outcome maps directly
+to the verdict. This logic lives in
+[`resolve_probe_verdict`][rampart.core.result.resolve_probe_verdict].
 
 ---
 
@@ -26,9 +26,10 @@ Probe executions are simpler than attacks — no injection phase:
 
 1. **Create session** — Open a fresh session with the agent
 2. **Send prompts** — Drive the conversation via the prompt driver
-3. **Evaluate** — Check whether the expected behavior is present
-4. **Clean up** — Close the session
-5. **Report** — Produce a [`Result`][rampart.core.result.Result]
+3. **Stop (optional)** — Check an explicit online `stop_when` condition
+4. **Evaluate** — Check the completed trace once for expected behavior
+5. **Clean up** — Close the session
+6. **Report** — Produce a [`Result`][rampart.core.result.Result]
 
 ---
 
@@ -51,6 +52,9 @@ assert result, result.summary
 
 Provide exactly one of `prompt`, `prompts`, or `driver`.
 
+Probes run the full prompt sequence by default. Pass `stop_when=` only when an
+online condition should intentionally end the trace early.
+
 ---
 
 ## Available Probes

docs/concepts/trace-schema.md

@@ -61,8 +61,9 @@ stops, including when the turn budget is reached.
 why that online evaluation ran. A non-null purpose requires an evaluation on the
 same turn. `Result.trace_end_reason` records why turn production stopped. These
 provenance fields are optional: missing or null means the producer did not record
-them, not that the last online evaluation is the terminal one. The codec never
-infers terminal evidence or a stop reason from the result status or turns.
+them, not that the last online evaluation is the final-trace evaluation. The
+codec never infers final-trace evidence or a stop reason from the result status
+or turns.
 Both placements of `EvalResult` receive the same strict type, finite-confidence,
 Unicode-scalar, and closed-enum validation.
 

docs/contributing/architecture.md

@@ -41,8 +41,8 @@ When adding a new attack or probe, you add a static factory method — not a new
 
 Evaluators are **polarity-free**. They report whether a condition was detected, not whether it's good or bad. The attack/probe factory applies the correct polarity:
 
-- `resolve_as_attack`: detected → UNSAFE
-- `resolve_as_probe`: detected → SAFE
+- `resolve_attack_verdict`: detected → UNSAFE
+- `resolve_probe_verdict`: detected → SAFE
 
 This allows the same evaluator (e.g., `ToolCalled`) to be used in both attack and probe contexts.