WIP: Integrate mini-SWE-agent as external baseline

CURRENT_STATE.md

@@ -3,28 +3,54 @@
 ## Goal
 Benchmark whether slice-isolated coding agents generalize from test-driven repair to typecheck-driven repair using the same projection/edit loop.
 
-## Active Slice
-Typecheck benchmark surface is implemented and wired end-to-end. Three plumbing fixtures pass `tsc --noEmit` failure through existing projectors, and results save separately. Awaiting first live benchmark run.
+## Status
+Typecheck benchmark is implemented, run, and producing clean results across 6 tasks (3 L1 + 3 L2). Results are recorded in `results/benchmark-typecheck.json`.
 
-## Inputs
-- `tsc --noEmit` output from task directory
-- benchmark config at `configs/benchmark-typecheck.json`
+mini-SWE-agent integration is wired end-to-end: adapter, runner, config, and CLI flags verified against v2.2.7. Ready for first benchmark run with `configs/benchmark-with-swe-agent.json`.
 
-## Expected Output
-- parsed `TypeErrorAnchor` with file/line/column/code/message
-- deterministic primary anchor selection
-- separate typecheck benchmark results at `results/benchmark-typecheck.json`
+## Benchmark Tasks
+
+### L1 (single-file, anchor file alone is sufficient)
+- `wrong_return_type` (TS2322) — all ZCA agents pass in 1 step
+- `missing_property` (TS2339) — all ZCA agents pass in 1 step
+- `undefined_name` (TS2304) — all ZCA agents pass in 1 step
+
+### L2 (cross-file, fix requires context beyond the anchor file)
+- `cross_file_return_type` (TS2322) — both ZCA agents pass; naive may be guessing
+- `wrong_method_call` (TS2339) — both ZCA agents pass; naive may be guessing
+- `unresolved_cross_import` (TS2304) — **clean naive/adaptive split**: naive FAIL, adaptive PASS
+
+## Latest Results (Sonnet 4)
+
+| Agent | Pass rate | Input tokens |
+|---|---|---|
+| Baseline | 0/6 | 3.9M in / 13.7k out |
+| ZCA Naive | 5/6 | 8.4k in / 4.0k out |
+| ZCA Adaptive | 6/6 | 5.9k in / 2.0k out |
+
+## Interpretation
+- Projection vs baseline is strongly supported on the typecheck surface
+- Typecheck generalization is strongly supported — same loops, same projectors, new signal
+- Adaptive vs naive is cleanly supported by `unresolved_cross_import`
+- Two of three L2 tasks are soft — naive passes by local guessing, not structural reasoning
+
+## Bug Fixed This Iteration
+The naive projector crashed with ENOENT on re-projection when `selectPrimaryAnchor` returned null (unsupported error code after model edit). The fallback path hit `inferSourceFile`, which is test-specific and defaults to `src/index.ts`. Fixed by:
+1. Adding anchor fallback: `primary?.file ?? anchors[0]?.file` in `ZCAAgent.ts`
+2. Adding `existsSync` guards in both `projectFailureSlice.ts` and `adaptiveProjector.ts`
 
 ## In Scope
 - `src/runtime/tools/runTypeCheck.ts` — signal execution + anchor parsing + selection
-- `src/agents/zca/ZCAAgent.ts` — signal routing, typecheck projector construction
+- `src/agents/zca/ZCAAgent.ts` — signal routing, typecheck projector construction, anchor fallback
 - `src/agents/baseline/BaselineAgent.ts` — signal routing for baseline
 - `src/agents/zca/zcaLoop.ts` — parameterized verify/prompt (shared by both signals)
 - `src/agents/baseline/baselineLoop.ts` — parameterized verify/prompt (shared by both signals)
-- `src/agents/zca/projectFailureSlice.ts` — entryFile override (no new projector)
-- `src/agents/zca/adaptiveProjector.ts` — entryFile override (no new projector)
-- `experiments/tasks-typecheck/` — 3 plumbing fixtures
+- `src/agents/zca/projectFailureSlice.ts` — entryFile override + existsSync guard
+- `src/agents/zca/adaptiveProjector.ts` — entryFile override + existsSync guard
+- `src/agents/sweAgent/MiniSWEAgentAdapter.ts` — subprocess adapter for mini-SWE-agent CLI
+- `experiments/tasks-typecheck/` — 6 fixtures (3 L1 + 3 L2)
 - `configs/benchmark-typecheck.json` — separate config
+- `configs/benchmark-with-swe-agent.json` — test benchmark with mini-SWE-agent baseline
 
 ## Out of Scope
 - generic plugin/signal framework
@@ -38,52 +64,20 @@ Typecheck benchmark surface is implemented and wired end-to-end. Three plumbing
 - supported error codes: TS2322, TS2339, TS2304 only
 - anchor selection ignores node_modules, dist, .d.ts files
 - one primary anchor per iteration (deterministic: sorted by file/line/code, first match)
+- anchor fallback to first raw anchor when supported-code filter is empty
 - projectors reused directly via entryFile override — no separate typecheck projector files
 - loops reused directly via verify/prompt parameterization — no separate loop files
-- 3 fixture tasks are plumbing-quality, not published benchmark coverage
-
-## Unknown Constraints
-- whether Sonnet/Haiku can reliably fix type errors with the current prompt framing
-- whether adaptive projector's import-following helps for type errors (may already be single-file)
-- whether 3 tasks are enough to expose architectural differences between agents
-- whether `tsc --noEmit` startup time affects duration metrics meaningfully
-
-## Verification
-- Command: `npm run benchmark -- configs/benchmark-typecheck.json`
-- Success condition: all 3 tasks run for all 3 agents, results save to `results/benchmark-typecheck.json`
-
-## Current Owners
-- signal execution + anchor parsing → `src/runtime/tools/runTypeCheck.ts`
-- anchor → projector routing → `src/agents/zca/ZCAAgent.ts`
-- naive projection → `src/agents/zca/projectFailureSlice.ts`
-- adaptive projection → `src/agents/zca/adaptiveProjector.ts`
-- ZCA loop → `src/agents/zca/zcaLoop.ts`
-- baseline loop → `src/agents/baseline/baselineLoop.ts`
-- benchmark orchestration → `src/scripts/runBenchmark.ts`
-- task sandboxing → `src/runtime/execution/sandbox.ts`
-- result types → `src/analysis/metrics/types.ts`
-
-## Files Changed (this iteration)
-- **new:** `src/runtime/tools/runTypeCheck.ts`
-- **new:** `configs/benchmark-typecheck.json`
-- **new:** `experiments/tasks-typecheck/wrong_return_type/` (TS2322)
-- **new:** `experiments/tasks-typecheck/missing_property/` (TS2339)
-- **new:** `experiments/tasks-typecheck/undefined_name/` (TS2304)
-- **modified:** `src/agents/zca/ZCAAgent.ts` — signal routing, typecheck projector builder
-- **modified:** `src/agents/zca/zcaLoop.ts` — optional verify/systemPrompt/goal/failureLabel
-- **modified:** `src/agents/zca/projectFailureSlice.ts` — optional entryFile parameter
-- **modified:** `src/agents/zca/adaptiveProjector.ts` — options object with entryFile
-- **modified:** `src/agents/zca/canonicalizeState.ts` — optional goal/failureLabel
-- **modified:** `src/agents/zca/zcaPrompt.ts` — uses failureLabel from state
-- **modified:** `src/agents/baseline/BaselineAgent.ts` — signal routing
-- **modified:** `src/agents/baseline/baselineLoop.ts` — optional verify/prompt/messages
-- **modified:** `src/runtime/tools/index.ts` — signal-aware tool registry
-- **modified:** `src/runtime/execution/taskPaths.ts` — optional tasksDir
-- **modified:** `src/runtime/execution/sandbox.ts` — propagate tasksDir
-- **modified:** `src/scripts/runBenchmark.ts` — parse signal/tasksDir from config
-- **modified:** `src/analysis/metrics/types.ts` — typecheck task classifications
-- **modified:** `src/runtime/execution/logger.ts` — verbose logging (from earlier)
-- **modified:** `src/scripts/runZCA.ts` — --verbose flag (from earlier)
+
+## Resolved Questions
+- **Will the baseline agent handle tsc errors?** No — same exploration-without-editing pattern as tests.
+- **Does adaptive projector's import-following help for type errors?** Yes — cleanly demonstrated on `unresolved_cross_import`.
+- **Are 3 tasks enough to expose differences?** No — 3 L1 tasks showed no naive/adaptive split. After adding 3 L2 tasks, one (`unresolved_cross_import`) cleanly separates them.
+- **Is `tsc --noEmit` startup time a problem?** No — each verification takes ~1s, negligible vs model latency.
+
+## Open Questions
+- How to design L2/L3 tasks where the naive projector reliably fails (not just sometimes)
+- Whether `extractFailingSymbol` should be adapted for tsc output (currently test-oriented)
+- Whether bounded parallel execution in the benchmark runner would meaningfully reduce wall-clock time
 
 ## Architecture Shape
 ```
@@ -119,11 +113,5 @@ Typecheck benchmark surface is implemented and wired end-to-end. Three plumbing
       └──────────────────────────────────┘
 ```
 
-## Open Questions
-- will the baseline agent's exploration loop handle `tsc` errors as effectively as test errors?
-- should the typecheck prompt include the specific error code and line for better targeting?
-- is `npx tsc --noEmit` startup latency acceptable or should we cache the compiler?
-- should future tasks include multi-file type errors (L2/L3 locality)?
-
 ## Next Minimal Step
-Run the typecheck benchmark end-to-end with `npm run benchmark -- configs/benchmark-typecheck.json` and record results. Then assess whether the projection architecture holds or needs adjustment before adding more tasks.
+Run the SWE-agent benchmark with `npm run benchmark -- configs/benchmark-with-swe-agent.json` and record results. Then design harder L2/L3 typecheck tasks where the naive projector reliably fails.

README.md

@@ -102,7 +102,7 @@ How obvious the bug source is from the failure signal.
 
 # Agents compared
 
-Three agents are evaluated.
+Four agents are evaluated.
 
 ## Baseline agent
 
@@ -122,6 +122,14 @@ Context grows across iterations. A "step" is counted only when the agent makes a
 
 ---
 
+## mini-SWE-agent (external baseline)
+
+An external coding agent used as a second baseline for comparison. [mini-SWE-agent](https://github.com/SWE-agent/mini-swe-agent) is a compact autonomous software-engineering agent from the SWE-bench ecosystem.
+
+It runs the same tasks, in the same sandboxes, evaluated by the same verifiers. The benchmark harness invokes it as a subprocess and collects metrics externally. This provides a recognizable external reference point without changing the benchmark architecture.
+
+---
+
 ## ZCA Naive
 
 Slice-isolated execution with a simple projector.
@@ -255,6 +263,15 @@ At the same time, some L2 tasks still appear soft or locally guessable, so the c
 
 ---
 
+# Current interpretation
+
+- **Projection vs baseline** is strongly supported across both test and typecheck surfaces. The baseline agent consistently fails by exploring without editing, regardless of failure signal.
+- **Typecheck generalization** is strongly supported. The same projection/edit loop, with no changes to projectors or agent loops, produces clean results on TypeScript compiler errors.
+- **Adaptive vs naive** is now cleanly supported by `unresolved_cross_import`, where the naive projector fails across 5 steps (it only sees the anchor file) while the adaptive projector solves it in 1 step (it follows imports to include context).
+- **More hard L2 tasks are still needed.** Two of three L2 tasks were solved by naive — likely because the model could guess the correct fix from the anchor file alone. Tasks where the fix is not locally inferrable are needed to further stress the naive/adaptive boundary.
+
+---
+
 # Key findings
 
 ### Execution architecture strongly affects agent behavior
@@ -342,6 +359,15 @@ If you want to run the compiler-driven benchmark:
 npm run benchmark -- configs/benchmark-typecheck.json
 ```
 
+To run the benchmark including mini-SWE-agent as an external baseline:
+
+```bash
+pip install mini-swe-agent
+npm run benchmark -- configs/benchmark-with-swe-agent.json
+```
+
+mini-SWE-agent is a Python CLI. It must be installed separately and available as `mini` on your PATH. The adapter invokes it as a subprocess and does not require any other Python dependencies in this repository.
+
 ---
 
 # Repository structure
@@ -364,6 +390,7 @@ experiments/tasks-typecheck/            # compiler-driven benchmark task definit
 src/
   agents/baseline/                      # long-context baseline agent
   agents/zca/                           # ZCA agent + naive/adaptive projectors
+  agents/sweAgent/                      # mini-SWE-agent subprocess adapter
   model/                                # model client abstraction
   runtime/                              # sandbox execution and tool registry
   analysis/                             # result types and metrics
@@ -373,6 +400,7 @@ configs/
   benchmark.json                        # same-model (Sonnet) test benchmark
   benchmark-cross-model.json            # cross-model (Opus vs Haiku) test benchmark
   benchmark-typecheck.json              # typecheck-driven benchmark
+  benchmark-with-swe-agent.json         # test benchmark with mini-SWE-agent baseline
 
 results/                                # benchmark output JSON files
 ```

configs/README.md

@@ -9,4 +9,8 @@ Configuration files for agent runs and benchmark matrix.
 | `baseline.json` | Config for baseline agent (max steps, model provider) |
 | `zca.json` | Config for ZCA agent with naive projector |
 | `zca-adaptive.json` | Config for ZCA agent with adaptive projector |
-| `benchmark.json` | Full benchmark matrix: tasks × agents × model |
+| `benchmark.json` | Same-model (Sonnet) test benchmark: tasks × agents × model |
+| `benchmark-cross-model.json` | Cross-model benchmark: Opus baseline vs Haiku ZCA agents |
+| `benchmark-typecheck.json` | TypeScript compiler-driven benchmark (all 6 typecheck tasks) |
+| `benchmark-typecheck-fast.json` | Typecheck benchmark subset (L2 tasks only, ZCA agents only) |
+| `benchmark-with-swe-agent.json` | Test benchmark with mini-SWE-agent as external baseline |

configs/benchmark-typecheck-fast.json

@@ -0,0 +1,18 @@
+{
+  "signal": "typecheck",
+  "tasksDir": "experiments/tasks-typecheck",
+  "tasks": [
+    "cross_file_return_type",
+    "wrong_method_call",
+    "unresolved_cross_import"
+  ],
+  "agents": [
+    { "name": "zca-naive", "type": "zca", "projector": "naive", "maxSteps": 5 },
+    { "name": "zca-adaptive", "type": "zca", "projector": "adaptive", "maxSteps": 5 }
+  ],
+  "model": {
+    "provider": "anthropic",
+    "model": "claude-sonnet-4-20250514",
+    "temperature": 0
+  }
+}

configs/benchmark-with-swe-agent.json

@@ -0,0 +1,19 @@
+{
+  "tasks": [
+    "parser_bug",
+    "range_check_bug",
+    "slug_conflict_bug",
+    "config_lookup_bug"
+  ],
+  "agents": [
+    { "name": "baseline", "type": "baseline", "maxSteps": 10 },
+    { "name": "mini-swe-agent", "type": "swe", "maxSteps": 10, "costLimit": 2.0, "timeout": 300000 },
+    { "name": "zca-naive", "type": "zca", "projector": "naive", "maxSteps": 5 },
+    { "name": "zca-adaptive", "type": "zca", "projector": "adaptive", "maxSteps": 5 }
+  ],
+  "model": {
+    "provider": "anthropic",
+    "model": "claude-sonnet-4-20250514",
+    "temperature": 0
+  }
+}

package.json

@@ -15,6 +15,7 @@
     "typescript": "^5.4.0"
   },
   "dependencies": {
-    "@anthropic-ai/sdk": "^0.79.0"
+    "@anthropic-ai/sdk": "^0.79.0",
+    "dotenv": "^17.3.1"
   }
 }

results/benchmark-typecheck.json

@@ -1,6 +1,6 @@
 {
   "results": [
-    {
+  {
       "task": "wrong_return_type",
       "agent": "baseline",
       "success": false,

results/benchmark-with-swe-agent.json

@@ -0,0 +1,20 @@
+{
+  "results": [
+    {
+      "task": "parser_bug",
+      "agent": "mini-swe-agent",
+      "success": true,
+      "steps": 0,
+      "durationMs": 120460,
+      "inputTokens": 0,
+      "outputTokens": 0
+    }
+  ],
+  "matrix": [],
+  "tasks": [
+    "parser_bug"
+  ],
+  "agents": [
+    "mini-swe-agent"
+  ]
+}