Skip to content

Time the phases of a test with E2E_DEBUG - #435

Merged
kblok merged 2 commits into
mainfrom
pr-c-debug-trace
Oct 18, 2026
Merged

kblok merged 2 commits into
mainfrom
pr-c-debug-trace

Conversation

@kblok

@kblok kblok commented Oct 17, 2026 •

Copy link
Copy Markdown
Member

Upstream's --debug times the phases of a run and keeps each agent step's transcript. The port had no timings, and it kept every transcript. Now E2E_DEBUG=1 makes each test print a table of its phase timings (launch, body, close, and the agent's observe, model, and action) to stderr, and an executor's transcript is kept only then, as upstream does. The agent step table needs the step recorder and comes later.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KWUdrjfarTEM4dmBbK6GaV

Summary by CodeRabbit

  • New Features
    • Enable E2E_DEBUG=1 or true to view per-test phase timings in stderr, including the slowest phases first. A fixture’s Debug setting can also enable this output.
    • Debug mode retains redacted executor transcripts alongside attempt results; transcripts are not retained when debug mode is off.
    • Documentation clarifies which phases are timed and notes that the agent-step timing table is not available.

kblok and others added 2 commits October 17, 2026 20:52
Upstream's --debug times the phases of a run and keeps each agent step's transcript. The port had neither, and kept transcripts always. E2E_DEBUG=1 now makes each test print its phase timings to stderr, and an executor's transcript is kept only then, as upstream does. The agent step table needs the step recorder and is left for later.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KWUdrjfarTEM4dmBbK6GaV
@coderabbitai

coderabbitai Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 2a43d6c3-e8de-412a-84ba-77b3efb354e1
📥 Commits

Reviewing files that changed from the base of the PR and between 41bb93d and 9501847.

📒 Files selected for processing: 19
  • COMPATIBILITY.md
  • README.md
  • skills/e2e-dotnet/SKILL.md
  • skills/e2e-dotnet/references/agent.md
  • skills/e2e-dotnet/references/debugging.md
  • skills/e2e-dotnet/references/running.md
  • src/E2E/Agent/ActContext.cs
  • src/E2E/Agent/ActTools.cs
  • src/E2E/Agent/ActionDispatcher.cs
  • src/E2E/Agent/Agent.cs
  • src/E2E/Agent/ObservationFeed.cs
  • src/E2E/Agent/Phases.cs
  • src/E2E/Agent/ToolLoop.cs
  • src/E2E/Internal/DebugTrace.cs
  • src/E2E/Run/E2EFixture.cs
  • src/E2E/Run/E2ESession.cs
  • tests/E2E.Tests/AgentExecutorContext/AttachedTranscriptTests.cs
  • tests/E2E.Tests/Debug/DebugTraceTests.cs
  • tests/E2E.Tests/Debug/SessionDebugTests.cs

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 1 remain after this review.

📝 Walkthrough

This change adds opt-in phase timing for .NET test sessions and agent operations. It writes timing summaries when sessions finish and retains attached executor transcripts only when debug mode is enabled. Documentation and tests describe and verify these behaviors.

Changes: Debug mode

Layer / File(s) Summary
Phase timing collection and agent instrumentation
src/E2E/Internal/DebugTrace.cs, src/E2E/Agent/Phases.cs, src/E2E/Agent/Agent.cs, src/E2E/Agent/ObservationFeed.cs, src/E2E/Agent/ActionDispatcher.cs, src/E2E/Agent/ActTools.cs, src/E2E/Agent/ToolLoop.cs, tests/E2E.Tests/Debug/DebugTraceTests.cs
Adds a collector that aggregates phase duration, count, and maximum duration. Agent model, observation, and action operations use the trace. Tests cover successful and failed timed work.
Session debug configuration and output
src/E2E/Run/E2EFixture.cs, src/E2E/Run/E2ESession.cs, tests/E2E.Tests/Debug/SessionDebugTests.cs, README.md, COMPATIBILITY.md, skills/e2e-dotnet/SKILL.md, skills/e2e-dotnet/references/agent.md, skills/e2e-dotnet/references/debugging.md, skills/e2e-dotnet/references/running.md
The fixture enables debug mode when E2E_DEBUG is 1 or true, and session options expose a debug setting. The session times launch, test-body, and close phases, then writes a summary to the configured writer or stderr. Tests and documentation cover the setting and output.
Debug-gated executor transcripts
src/E2E/Agent/ActContext.cs, tests/E2E.Tests/AgentExecutorContext/AttachedTranscriptTests.cs, COMPATIBILITY.md
Attached transcript text is stored only when debug mode is enabled. Tests check that transcripts are omitted when disabled and redacted when enabled.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant E2EFixture
  participant E2ESession
  participant DebugTrace
  participant Agent
  E2EFixture->>E2ESession: Pass Debug in session options
  E2ESession->>DebugTrace: Create trace from debug setting
  E2ESession->>Agent: Pass trace through attempt scope
  Agent->>DebugTrace: Record timed agent phases
  E2ESession->>DebugTrace: Record session phases and test body time
  E2ESession->>DebugTrace: Format and write summary
Loading
Merge Risk: ⚪ Minimal · up to 95018

The debug documentation is accurate, and no issue requiring a fix before merge was established.

Security architecture risk: ⚪ Minimal · up to 95018

Default execution retains less diagnostic text. Optional timing output preserves existing execution permissions, redaction, and resource ownership.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The inspected change affects diagnostic output from the test process and transcript files under configured attempt results. Enabling debugging restores the previous transcript-retention behavior rather than introducing a new transcript destination or broader execution authority.

Trust Boundaries and Controls

  • observed — Executor-supplied transcript text crosses into filesystem persistence through the attempt redactor. The existing terminal path closes the step budget and writes the transcript in a finally block; the PR adds the debug gate without changing that sink or redaction control.

Resilience and Maintainability Implications

  • observed — Transcript writes continue to tolerate filesystem access failures. Timing output suppresses IOException, but not every possible writer exception; such a failure can affect teardown reporting after engine and cancellation-resource cleanup has already run.
Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 46.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 30 functions across 13 files. (6 skipped:… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the main change: adding phase timing for tests through E2E_DEBUG.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 46.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 30 functions across 13 files. (6 skipped: 6 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

  • Fix all pre-merge checks with AI
✨ Finishing Touches: 💡 1
📝 Generate docstrings: 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@kblok
kblok merged commit 7a49de3 into main Oct 18, 2026
2 checks passed
@kblok
kblok deleted the pr-c-debug-trace branch October 18, 2026 02:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant