fix(agent): append system_prompt to the Claude Code preset instead of replacing

docs/AB_EXPERIMENTS.md

@@ -136,6 +136,11 @@ fair game: `model`, `permission_mode`, `allowed_tools`, `disallowed_tools`,
 `plugins`, `system_prompt` / `system_prompt_file`, `setting_sources`,
 `claude_settings`, `sdk_options`.
 
+> **`system_prompt_mode` is a poor A/B lever.** A `replace` arm sends no default
+> Claude Code prompt *at all*, so the delta measures the missing behavioral guidance
+> (tool-call batching, conciseness), not your prompt text. To A/B prompt *content*,
+> vary `system_prompt` and leave both arms on `append`.
+
 > **Path-resolution gotcha.** Relative file paths in variant config resolve
 > against _different_ base directories depending on the field:
 >

docs/REPORT_SCHEMA.md

@@ -137,6 +137,14 @@ The authoritative per-replicate record.
 `ClaudeAgentOptions` dump), `sandbox_path`, `task_config`
 (`{resolved, source_yaml, source_file, lineage}` — `lineage` maps each field to
 `{value, source, source_detail}` so you can trace which config layer set it).
+`environment_info.system_prompt_semantics` (`"append"` / `"replace"`) records the
+system-prompt regime the agent ran with; runs predating the marker used
+replace-on-set / empty-on-unset semantics and are not score-comparable, so
+consumers should segment on it (absent key ⇒ pre-append regime).
+`sdk_options.system_prompt` is a `SystemPromptPreset` dict
+(`{type: "preset", preset: "claude_code", exclude_dynamic_sections: true, append?: str}`)
+on append-mode Claude Code runs and a plain string only in replace mode — it is
+no longer `str | null`, so consumers must not string-handle it unconditionally.
 
 **Telemetry/totals:** `total_token_usage` ([TokenUsage](#tokenusage)),
 `command_stats` (`CommandStatistics`), `total_assistant_turns`, `expected_commands` /

docs/agents/ANTIGRAVITY.md

@@ -109,6 +109,14 @@ Antigravity exposes a `thinking_level` field (`minimal` / `low` / `medium` /
 Antigravity-specific — Claude Code and Codex don't take this field. Thinking tokens
 are billed as **output** tokens (see [Telemetry](#telemetry)).
 
+### `system_prompt`
+
+`agent.system_prompt` is passed to the SDK as `system_instructions`, whose string
+shorthand maps to `TemplatedSystemInstructions` — a named section **appended** to
+the harness's default system instructions, never a replacement. This matches the
+append-only semantics of the shared config field across agents (Claude Code appends
+via the `claude_code` preset; Codex via `developer_instructions`).
+
 ### Skills (SKILL.md)
 
 Antigravity supports [Agent Skills](https://agentskills.io/specification)

docs/agents/CLAUDE_CODE.md

@@ -99,8 +99,9 @@ agent:
 | `allowed_tools` | `list[str] \| null` | Tool allowlist. Unset ⇒ all tools allowed. |
 | `disallowed_tools` | `list[str] \| null` | Tool denylist. (`ToolSearch` is always appended for Bedrock parity.) |
 | `plugins` | `list[{type: local, path}]` | Local plugin/skill directories; `$VAR` in `path` is expanded and resolved to an absolute path. |
-| `system_prompt` | `str \| null` | **Replaces** the default system prompt (there is no *append* seam). Mutually exclusive with `system_prompt_file`. |
-| `system_prompt_file` | `str \| null` | Path (relative to the task YAML) loaded into `system_prompt` at resolution. |
+| `system_prompt` | `str \| null` | **Appended** to the default Claude Code system prompt (via the SDK's `claude_code` preset) — the default's behavioral guidance is kept unless `system_prompt_mode: replace` opts out. Mutually exclusive with `system_prompt_file`. An empty or whitespace-only value is treated as unset. |
+| `system_prompt_mode` | `"append"` (default) / `"replace"` | `replace` sends `system_prompt` as the **entire** system prompt (no preset) and requires a non-blank `system_prompt` / `system_prompt_file` (validated at load). Used by judge sub-agents and the user simulator, which must not carry the coding-agent persona; rarely needed in tasks — see [the migration note](#migrating-tasks-that-set-system_prompt). |
+| `system_prompt_file` | `str \| null` | Path (relative to the task YAML) loaded into `system_prompt` at resolution. Works with either `system_prompt_mode`. |
 | `setting_sources` | `list["user"\|"project"\|"local"] \| null` | Which host setting sources the SDK reads. Default resolves to `["project"]`. See [Sandbox isolation](#sandbox-isolation). |
 | `claude_settings` | `str \| dict \| null` | Passed to the SDK `--settings`. A dict is JSON-serialized; a str is a settings file path. Use `permissions.deny` to block tools/paths. |
 | `sdk_options` | `dict` (default `{}`) | Pass-through for `ClaudeAgentOptions` fields Coder Eval doesn't own (e.g. `effort`). Validated at load — an unknown or framework-owned key is a hard error. |
@@ -111,6 +112,47 @@ agent:
 > `setting_sources`, `include_partial_messages`, …) are rejected there — set those
 > through their typed fields or `-D run_limits.*`. MCP servers are not a YAML field.
 
+> **System-prompt reproducibility.** In `append` mode the preset's *dynamic
+> sections* (working directory, git status, auto-memory) are excluded so the system
+> prompt stays identical across runs — the per-run sandbox tempdir path would
+> otherwise be baked into it, breaking prompt caching and run comparability. The
+> SDK re-injects the stripped content into the first user message, so the agent
+> loses nothing. Note the default-prompt baseline tracks the installed Claude Code
+> CLI version; `environment_info.claude_code_cli` in `run.json` records which
+> version a run used, and `environment_info.system_prompt_semantics`
+> (`append` / `replace`) records the prompt regime — runs predating that marker
+> used replace-on-set / empty-on-unset semantics and are not score-comparable.
+
+### Migrating tasks that set `system_prompt`
+
+`system_prompt` used to **replace** Claude Code's default system prompt. It now
+appends to it. If your task or experiment sets `system_prompt`, the agent gains back
+every default behavioral instruction it was previously running without — parallel
+tool-call batching, conciseness rules, the `Read`/`Grep`/`Glob` tool preferences, and
+the default security guardrails.
+
+That is a genuine behavior change, so **scores are not comparable across this
+boundary**. Pick one:
+
+- **Keep the repair (recommended).** Do nothing. Re-baseline any threshold or
+  reference score the task gates on, and expect turn counts to drop on tasks that
+  depend on batched tool calls.
+- **Preserve the old behavior.** Add `system_prompt_mode: replace` to the `agent:`
+  block. The configured prompt again becomes the entire system prompt. Only do this
+  if the task *intends* to run without the default guidance — a prompt that merely
+  adds sandbox policy or a persona does not.
+
+Segment dashboards on `environment_info.system_prompt_semantics` to keep the two
+regimes in separate cohorts. Note the append-mode baseline also tracks the installed
+CLI version, so a `CLAUDE_CODE_VERSION` bump becomes a score-affecting change
+(attributable via `environment_info.claude_code_cli`).
+
+> **Consumers reading `sdk_options.system_prompt`.** The persisted value changed
+> shape: a plain `str` in the old regime, a `SystemPromptPreset` dict
+> (`{type, preset, exclude_dynamic_sections, append}`) in append mode. Code that
+> string-handles that field needs to branch on the type — see
+> [Report schema](../REPORT_SCHEMA.md).
+
 ### Setting fields from the CLI
 
 Any of these merge-resolve through `-D` / `--set` (see

docs/agents/CODEX.md

@@ -213,6 +213,7 @@ The Codex SDK is synchronous. The agent uses `_run_async()` helper to detect and
 | **SDK Type** | Subprocess (CLI via JSON generator) | Sync client (app-server subprocess) |
 | **Command Tracking** | Full telemetry (tool name, params, duration) | Streamed telemetry: shell → `Bash`, apply_patch → `Write` |
 | **Model Selection** | Direct via `--model` or config | `agent.model` pinned into `thread_start` |
+| **System prompt** | `system_prompt` appended to the default prompt (SDK `claude_code` preset) | `system_prompt` passed as `developer_instructions` on top of the Codex base prompt |
 | **Session Resume** | `--resume {session_id}` | Via thread ID |
 | **Permissions** | `permission_mode` + `allowed_tools` | `permission_mode` → sandbox/approval + `allowed_tools`/`disallowed_tools` → thread config |
 | **Tool Enforcement** | Not enforced by Coder Eval wrapper | `enabled_tools` honored; `disabled_tools` NOT enforced by the SDK |
@@ -226,6 +227,20 @@ The Codex SDK is synchronous. The agent uses `_run_async()` helper to detect and
 4. **Authentication** - Requires `CODEX_API_KEY` in the environment (point it at whichever endpoint's key you use — OpenAI, gateway, or Azure); the agent calls `login_api_key` when a key is present. `OPENAI_API_KEY`/`AZURE_OPENAI_API_KEY` are NOT read.
 5. **Model field** - `TurnRecord.model_used` reflects the pinned `agent.model`; the Codex `Turn` payload itself doesn't carry the resolved model.
 6. **Skills with Windows paths** - Symlink creation may fail on Windows; agent falls back to copying (slower).
+7. **No `system_prompt_mode`** - `replace` semantics are Claude-Code-only. `system_prompt` is always appended as `developer_instructions`; setting `system_prompt_mode` on a Codex `agent:` block is a validation error (unknown field).
+
+## Migrating tasks that set `system_prompt`
+
+`system_prompt` was previously **ignored** on Codex tasks — silently dropped, so the
+task ran on Codex's base prompt alone. It is now forwarded as
+`developer_instructions`, layered on top of that base prompt. Any Codex task setting
+the field now actually receives those instructions, so **scores are not comparable
+across this boundary**. Two things to check:
+
+- A prompt written for Claude (naming `Read`/`Grep`/`Glob`, or Claude tool etiquette)
+  is now live on Codex, where those tool names don't exist.
+- There is **no opt-out** (see Known Limitations #7). To restore the old behavior,
+  remove `system_prompt` from the Codex variant — otherwise re-baseline.
 
 ## Future Enhancements
 

src/coder_eval/agents/antigravity_agent.py

@@ -348,7 +348,10 @@ async def start(
                 # Autonomous execution: approve every tool call (incl. run_command),
                 # which the default LocalAgentConfig policy would otherwise deny.
                 policies=[policy.allow_all()],
-                system_instructions=self.config.system_prompt or None,
+                # Forward verbatim (None stays None): an explicit "" is a
+                # configured-but-empty prompt and must be forwarded, matching
+                # the `is not None` semantics in the Claude Code / Codex agents.
+                system_instructions=self.config.system_prompt,
                 # Skill discovery: hand the harness the search-path roots that parent
                 # the UiPath skill dirs. Unlike Codex (which symlinks into
                 # .agents/skills/), Antigravity takes skill search paths natively.
@@ -561,6 +564,9 @@ def get_environment_info(self) -> dict[str, Any]:
         return {
             "antigravity_model": self._effective_model(),
             "antigravity_thinking_level": self.config.thinking_level,
+            # Antigravity has always appended (TemplatedSystemInstructions);
+            # emitted for cross-agent uniformity of the marker.
+            "system_prompt_semantics": "append",
         }
 
     def _conversation_or_none(self) -> Any:

src/coder_eval/agents/claude_code_agent.py

@@ -10,7 +10,7 @@
 from contextlib import suppress
 from datetime import datetime, timedelta
 from pathlib import Path
-from typing import Any, ClassVar
+from typing import Any, ClassVar, Literal
 
 from claude_agent_sdk import (
     ClaudeAgentOptions,
@@ -28,6 +28,10 @@
 # its kill target and timeouts will no longer be enforced at the agent layer.
 from claude_agent_sdk._internal.transport.subprocess_cli import SubprocessCLITransport
 
+# SystemPromptPreset is not re-exported from the SDK root, so claude_agent_sdk.types
+# is the only import route (same treatment as evaluation/verdict_tool.py).
+from claude_agent_sdk.types import SystemPromptPreset
+
 from coder_eval.agent import Agent, AgentState
 from coder_eval.agents._logging import PrefixedAdapter, log_raw_sdk_event
 from coder_eval.agents.registry import AgentRegistry
@@ -1173,6 +1177,26 @@ def _build_claude_query(
         if "ToolSearch" not in disallowed_tools:
             disallowed_tools.append("ToolSearch")
 
+        # The SDK maps system_prompt=None to `--system-prompt ""` (an explicit
+        # EMPTY custom prompt) and a plain string to a full replacement — either
+        # way Claude Code's default behavioral guidance (parallel tool-call
+        # batching, conciseness) is lost. So ALWAYS send the claude_code preset:
+        # without `append` the CLI runs its default prompt; with it the configured
+        # prompt is appended. exclude_dynamic_sections keeps the prompt static
+        # across runs (the per-run tempdir path would otherwise be baked into the
+        # system prompt, breaking prompt caching and run comparability); the SDK
+        # re-injects the stripped sections into the first user message.
+        # system_prompt_mode="replace" (judge sub-agents) opts out of the preset:
+        # the configured prompt IS the entire system prompt.
+        system_prompt: str | SystemPromptPreset
+        if self._effective_prompt_mode() == "replace":
+            assert self.config.system_prompt is not None  # guaranteed by _effective_prompt_mode
+            system_prompt = self.config.system_prompt
+        else:
+            system_prompt = SystemPromptPreset(type="preset", preset="claude_code", exclude_dynamic_sections=True)
+            if self.config.system_prompt is not None:
+                system_prompt["append"] = self.config.system_prompt
+
         # as_posix(), not str(): bash on Windows strips backslashes from unquoted
         # paths, so a redirect like `> D:\foo\bar` ends up writing to "Dfoobar".
         options = ClaudeAgentOptions(
@@ -1192,7 +1216,7 @@ def _build_claude_query(
             # summing per-message values undercounts by 10x+. Without this flag
             # StreamEvents are suppressed by the SDK.
             include_partial_messages=True,
-            system_prompt=self.config.system_prompt,
+            system_prompt=system_prompt,
             setting_sources=self.config.setting_sources if self.config.setting_sources is not None else ["project"],
             resume=self._session_id,
             settings=json.dumps(self.config.claude_settings)
@@ -1216,6 +1240,37 @@ def _build_claude_query(
 
         return options, transport, effective_model
 
+    def _effective_prompt_mode(self) -> Literal["append", "replace"]:
+        """The system-prompt regime that actually goes on the wire.
+
+        Single source of truth for both the options builder and the
+        ``system_prompt_semantics`` run-record marker, so the persisted regime
+        can never disagree with what was sent. ``replace`` requires a configured
+        prompt (the config validator rejects the pair at load, but a mutated or
+        hand-built config falls back to the preset here — fail open to append).
+        """
+        if self.config.system_prompt_mode == "replace":
+            if self.config.system_prompt is not None:
+                return "replace"
+            # Warn on the downgrade so it's visible in task.log rather than only
+            # inferable from run.json's system_prompt_semantics marker.
+            logger.warning(
+                "system_prompt_mode='replace' with no system_prompt — falling back to the claude_code "
+                + "preset (append regime). run.json records the regime actually used."
+            )
+        return "append"
+
+    def get_environment_info(self) -> dict[str, Any]:
+        """Record which system-prompt regime built this run's prompts.
+
+        ``append`` = the claude_code preset (dynamic sections excluded) with the
+        configured system_prompt, if any, appended; ``replace`` = the configured
+        prompt is the ENTIRE system prompt (judge sub-agents). Runs from before
+        this marker existed used replace-on-set / empty-on-unset semantics —
+        trend dashboards must not pool scores across that boundary.
+        """
+        return {"system_prompt_semantics": self._effective_prompt_mode()}
+
     async def stop(self) -> None:
         """Stop the agent and clean up resources."""
         self.client = None

src/coder_eval/agents/codex_agent.py

@@ -937,17 +937,25 @@ def _close_client(self) -> None:
     def get_environment_info(self) -> dict[str, Any]:
         """Record the resolved Codex routing so runs are auditable/comparable.
 
-        Only emits when a custom endpoint is configured (CODEX_BASE_URL). On a
-        custom endpoint the model is an operator-chosen alias (a deployment name
-        on Azure), so two operators' ``gpt-5-codex`` deployments are otherwise
-        indistinguishable in run artifacts. The host (not the full URL) is
-        recorded to avoid leaking any embedded credentials; the API key is never
-        recorded.
+        Always emits ``system_prompt_semantics``. The routing keys
+        (``codex_base_url_host`` / ``codex_wire_api`` / ``codex_api_version`` /
+        ``codex_model_is_deployment``) are added only when a custom endpoint is
+        configured (CODEX_BASE_URL): on a custom endpoint the model is an
+        operator-chosen alias (a deployment name on Azure), so two operators'
+        ``gpt-5-codex`` deployments are otherwise indistinguishable in run
+        artifacts. The host (not the full URL) is recorded to avoid leaking any
+        embedded credentials; the API key is never recorded.
         """
+        # system_prompt_semantics: Codex appends system_prompt as
+        # developer_instructions on top of its base prompt. Runs from before this
+        # marker existed silently DROPPED the field — dashboards must not pool
+        # system_prompt-setting tasks across that boundary.
+        info: dict[str, Any] = {"system_prompt_semantics": "append"}
         base_url = self._resolve_base_url()
         if not base_url:
-            return {}
+            return info
         return {
+            **info,
             "codex_base_url_host": urlparse(base_url).hostname or "",
             "codex_wire_api": _CODEX_WIRE_API,
             "codex_api_version": self._resolve_api_version() or "",
@@ -1276,6 +1284,14 @@ def _build_thread_options(self) -> dict[str, Any]:
             options["model"] = effective_model
             self._log.debug(f"Codex model pinned to {effective_model}")
 
+        # system_prompt maps to developer_instructions: injected ON TOP of Codex's
+        # base prompt, matching the append-only contract of the shared config field
+        # (Claude Code appends via the claude_code preset; Antigravity via
+        # TemplatedSystemInstructions). base_instructions (full replacement of the
+        # base prompt) is deliberately not exposed.
+        if self.config.system_prompt is not None:
+            options["developer_instructions"] = self.config.system_prompt
+
         permission_mode = self.config.permission_mode.value
         approval_mode_str = _CODEX_APPROVAL_MODE