Agents window: Compare multiple implementation attempts

[meganrogge]

Compare multiple implementation attempts in the Agents window

The Agents window supports multi-chat sessions, peer-chat forks, isolated worktrees, and reviewing changes beside a conversation. However, after exploring more than one approach, users must manually inspect each chat and diff to determine which implementation is better.

Add a Compare Attempts workflow that runs or selects multiple approaches to the same task and presents their outcomes in a code-oriented comparison.

Proposed experience

Add a Compare Agents toggle to the new-session input alongside the existing harness and model controls.

• When the toggle is off, the current single-harness behavior is unchanged.
• When the toggle is on, the harness picker becomes multi-select and the send action indicates how many attempts will run, for example Run 3 Attempts.
• The model remains a single shared selection. Only harnesses that advertise that model are eligible; incompatible harnesses are disabled with an explanation.
• The prompt, workspace, starting branch or commit, permission level, and isolation configuration are shared across all attempts.

Submitting the prompt fans it out to one new session per selected harness. The sessions are placed in a comparison group so they can be tracked together, while a failure to start one harness does not discard successful attempts.

Each attempt:

• receives the same prompt and starts from the same repository state;
• runs in a separate session and isolated worktree;
• uses one selected harness and the shared model;
• remains independently reviewable and editable.

Once at least two attempts have results, VS Code offers a comparison view containing:

• a concise implementation summary for each attempt;
• changed files and diff size;
• test, build, lint, and diagnostic results;
• unresolved errors or warnings;
• elapsed time and usage;
• notable architectural or behavioral differences;
• overlapping, conflicting, and attempt-specific changes.

The user can then:

• Use attempt A or Use attempt B;
• open the diffs side by side;
• send feedback to one attempt;
• ask an agent to Synthesize the best of both into a new attempt;
• discard the unused worktrees.

Why this belongs in the Agents window

This makes existing multi-session and worktree support useful as a deliberate workflow rather than just additional tabs. The Agents window can provide a stronger experience than a generic best-of-N response because it understands source control state, diagnostics, tests, diffs, and editor navigation.

It also gives users a concrete way to use parallel agents without introducing a separate multi-session dashboard.

Related approaches

Cursor 2.2 introduced multi-agent judging, which evaluates parallel agent runs after they finish, recommends a solution, and explains the selection. This validates the need for a decision step on top of parallel execution. The VS Code opportunity is to make the underlying code evidence inspectable and support user-controlled synthesis rather than stopping at a winner recommendation.

• Cursor changelog: Multi-agent judging

OpenAI Codex has a Best of N feature that generates multiple responses for one task so users can explore alternatives and choose an approach. The Codex app also supports parallel agents in isolated worktrees and reviewing and commenting on each agent's diff.

This proposal is not a request for exact parity. VS Code can provide a more code-native comparison by combining its multi-chat sessions, worktrees, diff editors, diagnostics, test results, and usage data into one decision surface. The distinguishing workflow is comparing validated implementation outcomes, then selecting or synthesizing them without manually reconciling separate chats.

• Codex changelog: Best of N
• Introducing the Codex app

Suggested initial scope

An initial version could:

1. Add comparison mode to the new-session input.
2. Let the user select two or more compatible harnesses and one shared model.
3. Fan out the same request into a separate worktree-isolated session for each harness.
4. Keep the resulting sessions together as a comparison group.
5. Compare summaries, changed files, diff statistics, diagnostics, validation results, elapsed time, and usage.
6. Open the chosen attempt's diff and worktree.

Automatic synthesis, applying the winner, and worktree cleanup could follow later.

Acceptance criteria

• One submission can launch two or more harness-specific sessions for comparison.
• VS Code clearly indicates whether their starting repository state differs.
• Validation results distinguish passed, failed, not run, and unknown.
• Selecting an attempt does not silently overwrite the user's current working tree.
• Unselected attempts remain available until the user explicitly discards them.
• The workflow is keyboard accessible and supports any selected harness that provides the shared model and isolated worktrees.

[TylerLeonhardt]

Related: #329648 tracks making create_session provider-aware, including explicit provider selection for Claude and Codex. This is related to this proposal’s multi-harness fan-out, where each comparison attempt targets a selected harness.