Skip to content

[REFACTOR]: Add shared linear trace runner - #148

Open
Spencer Schoenberg (spencrr) wants to merge 3 commits into
microsoft:mainfrom
spencrr:dev/spencrr/trace-runner
Open

Spencer Schoenberg (spencrr) wants to merge 3 commits into
microsoft:mainfrom
spencrr:dev/spencrr/trace-runner

Conversation

@spencrr

@spencrr Spencer Schoenberg (spencrr) commented Aug 4, 2026 •

Copy link
Copy Markdown
Contributor

Description

Adds the linear trace runner shared by attack and probe strategies. run_trace_async drives a conversation with an optional online stop_when check, keeps separate raw and annotated turn histories, and records TraceEndReason. evaluate_final_trace_async evaluates the final trace once. It reuses the latest online evaluation only when the evaluator, raw turns, manifest, and observability level are all identical.

The runner does not own session lifetime, polarity, cleanup, or exception conversion; those remain strategy and BaseExecution responsibilities. Driver history receives a shallow copy, evaluator contexts contain annotation-free turns, and reused evaluator evidence is copied defensively.

EvaluationRecord, TraceRun, run_trace_async, and evaluate_final_trace_async are exported from rampart.core for custom strategies and documented in the API reference. Built-in probes adopt them in #149 and XPIA in #150.

Breaking changes

None. This PR adds APIs and does not change existing strategies.

Checklist

  • pre-commit run --all-files passes
  • Tests added or updated for changes — termination reasons, raw and annotated histories, stop behavior, exact-context reuse, changed observability and manifest, optional-evidence copying, post-run mutation, exceptions, and turn budgets
  • Documentation updated

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@spencrr
Spencer Schoenberg (spencrr) force-pushed the dev/spencrr/trace-runner branch 2 times, most recently from a085b65 to cd2667c Compare August 27, 2026 17:30
@spencrr
Spencer Schoenberg (spencrr) force-pushed the dev/spencrr/trace-runner branch 2 times, most recently from c6e22c8 to 1fd45e0 Compare September 8, 2026 22:46
@spencrr
Spencer Schoenberg (spencrr) force-pushed the dev/spencrr/trace-runner branch 3 times, most recently from 560cb32 to 9faf7bc Compare September 29, 2026 22:20
Require the same observability and manifest before reusing an online judgment. Copy optional evidence through the shared tolerant renderer so malformed supporting text cannot discard an established verdict. Keep terminal evidence and operand lists independent from online records.
Rename evaluate_terminal_async to evaluate_final_trace_async so the public runner helper matches Result.final_trace_evaluation. Document the trace execution helpers where they are introduced.
@spencrr
Spencer Schoenberg (spencrr) marked this pull request as ready for review October 1, 2026 05:16
@spencrr
Spencer Schoenberg (spencrr) requested review from a team and a balanced review from Copilot October 1, 2026 05:16
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The implementation matches the documented contract and has thorough focused test coverage.

Review effort: Balanced
Findings: None

What changed in this PR

Adds a shared linear trace runner for future attack and probe strategies, including final-trace evaluation reuse.

Changes:

  • Adds trace execution, termination tracking, and final evaluation helpers.
  • Exports and documents the new core APIs.
  • Adds comprehensive unit coverage for stopping, histories, reuse, mutation, and budgets.
File Description
rampart/​core/​trace.py Implements shared trace execution and evaluation reuse.
rampart/​core/​__init__.py Exports the new public APIs.
docs/​api/​core-protocols.md Adds API reference entries.
tests/​unit/​core/​test_trace.py Tests trace execution and reuse behavior.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants