From cc0c6da4fd2677144355c827c3d39fc796c89d8c Mon Sep 17 00:00:00 2001 From: Harry Yep Date: Tue, 13 Oct 2026 17:25:49 +0800 Subject: [PATCH 1/8] feat(codex): verify the companion on codex-cli 0.162.1 and classify flex capacity and stock refusals --- docs/codex-contract.md | 6 ++++-- plugins/codex/scripts/codex-companion.mjs | 2 +- plugins/codex/scripts/lib/codex-exec.mjs | 8 ++++++-- plugins/fusion/verified-versions.json | 4 ++-- tests/breaker-check.test.mjs | 4 ++-- tests/codex-companion.test.mjs | 12 ++++++------ tests/codex-exec.test.mjs | 6 +++--- tests/failure-kinds.test.mjs | 23 +++++++++++++++++++++++ tests/fake-codex | 2 +- tests/verified-versions.test.mjs | 2 +- 10 files changed, 49 insertions(+), 20 deletions(-) diff --git a/docs/codex-contract.md b/docs/codex-contract.md index 892301f..64271cd 100644 --- a/docs/codex-contract.md +++ b/docs/codex-contract.md @@ -61,6 +61,8 @@ Every Codex exec invocation disables `multi_agent`, `multi_agent_v2`, `sleep_too The server model catalog now sets `multi_agent_version`, which `multi_agent_version_for_model` prefers over the `multi_agent` feature flags unless `agents.enabled` is false (`codex-rs/core/src/config/mod.rs` at rust-v0.157.1). The companion therefore passes `agents.enabled=false` and disables `computer_use`, `browser_use`, `browser_use_external`, `in_app_browser`, `image_generation`, `apps`, `plugins`, and `remote_plugin` to remove delegated agents, connector, plugin, computer use, browser, and image generation tools. A live exec on codex-cli 0.157.1 exposed only exec, wait, and request_user_input while workspace-write still emitted file changes and command execution events. +The rust-v0.162.1 source re-verification confirmed that all 13 disabled feature keys and every `-c` override the companion passes remain valid. `write_stdin_approval` is now stable and enabled by default; the companion keeps it under `approval_policy="never"`. API key model discovery is also stable and enabled by default, so API key runs may fetch an account model catalog independently of the tool sandbox network toggle. `--cyber-access-program` is opt in and rejects native review. A failed explicit provider catalog fetch can leave `models_cache.json` with an empty models array. + Guardian approval reviews cannot run inside a companion job. Upstream routes an action to Guardian only under the on request or granular approval policies with the auto review reviewer, and the companion pins `approval_policy="never"`, so the hidden `codex-auto-review` catalog model and the `node_repl_auto_review_required` flag that GPT-6 Astra alone carries have no effect on delegated runs (guardian routing read at rust-v0.153.4 and re-read unchanged at rust-v0.154.0 and main 5b1d656, where `routes_approval_policy_to_guardian` still requires the on request or granular policy together with the auto review reviewer). Skills stay under the user's Codex configuration. The companion does not set `skills.include_instructions`, so a delegated run may pay the context cost of the enabled skill catalog, and when Codex trims that catalog it reports `Skill descriptions were shortened to fit the skills context budget` as an `item.error` diagnostic that the job record retains. The remedy is user side: the `skills.max_context_tokens` key or disabling unused skills in `config.toml`. @@ -103,13 +105,13 @@ Successful transport output includes `state: done`. Error and cancelled output i ## Failure kinds -The adapter uses the shared failure categories `quota`, `auth`, `missing_cli`, `setup`, `rate_limited`, `timeout`, `error`, `permission`, `cancelled`, `input`, `died`, and `resource`. Protocol violations use `protocol`. A process exit caused by an unrequested signal uses `process`. A forbidden collaboration tool call uses `policy`. A server side refusal (misalignment, cyber, or content policy) uses `refused`: Codex exec delivers it only as the `turn.failed` message text, without the structured `codex_error_info` that the app-server protocol carries, so the classification matches the message; `refused` reflects the brief rather than lane health, and the circuit breaker ignores it. +The adapter uses the shared failure categories `quota`, `auth`, `missing_cli`, `setup`, `rate_limited`, `timeout`, `error`, `permission`, `cancelled`, `input`, `died`, and `resource`. Protocol violations use `protocol`. A process exit caused by an unrequested signal uses `process`. A forbidden collaboration tool call uses `policy`. A server side refusal (misalignment, cyber, or content policy) uses `refused`: Codex exec delivers it only as the `turn.failed` message text, without the structured `codex_error_info` that the app-server protocol carries, so the classification matches the message; `refused` reflects the brief rather than lane health, and the circuit breaker ignores it. The 0.162.1 content filter message and stock cybersecurity and biological risk refusal messages classify as `refused`. The Flex capacity message classifies as `rate_limited`. Classification uses structured Codex events first, then the process error and bounded stderr tail. Logs may retain more detail, but rendered failures expose only bounded diagnostics and never expose environment variables or authentication data. ## Compatibility -The companion checks the installed Codex version during setup and records it on every job when available. Preflight hard-fails with `failureKind: "setup"` when the installed Codex CLI version parses below the tested minimum 0.154.0 and tells the user to upgrade. The tested interval runs from 0.154.0 up to but excluding 0.158.0; versions below 0.154.0 fail task and review preflight with a setup error; versions at or above 0.158.0 keep running task and review preflight while `/codex:setup` reports not ready with exit code 1 as the compatibility advisory. Live verification on 2026-09-05 confirmed task execution (including a live web search consult on gpt-6-astra at 0.153.2), thread resume by stored thread id, structured output via `--output-schema`, native review, and JSONL rollout persistence on codex-cli 0.153.2 and again on 0.153.4; on 2026-09-14 a read only consult, thread resume by stored thread id, structured output via `--output-schema`, and native review passed on codex-cli 0.154.0. Setup also verifies CLI authentication and a writable adapter data directory. A nonempty `CODEX_API_KEY` is accepted as exec authentication when `codex login status` reports logged out because that probe does not inspect environment based authentication. Unknown JSONL event types are retained and ignored unless they affect terminal correctness. Missing or changed required lifecycle events fail closed as protocol errors. +The companion checks the installed Codex version during setup and records it on every job when available. Preflight hard-fails with `failureKind: "setup"` when the installed Codex CLI version parses below the tested minimum 0.154.0 and tells the user to upgrade. The tested interval runs from 0.154.0 up to but excluding 0.163.0; versions below 0.154.0 fail task and review preflight with a setup error; versions at or above 0.163.0 keep running task and review preflight while `/codex:setup` reports not ready with exit code 1 as the compatibility advisory. Live verification on 2026-09-05 confirmed task execution (including a live web search consult on gpt-6-astra at 0.153.2), thread resume by stored thread id, structured output via `--output-schema`, native review, and JSONL rollout persistence on codex-cli 0.153.2 and again on 0.153.4; on 2026-09-14 a read only consult, thread resume by stored thread id, structured output via `--output-schema`, and native review passed on codex-cli 0.154.0; on 2026-10-13 setup reported the build as tested, structured output via `--output-schema`, thread resume by stored thread id, and native review passed on codex-cli 0.162.1, and nine gpt-6-luna write tasks finished `done` with their task verifiers passing. Setup also verifies CLI authentication and a writable adapter data directory. A nonempty `CODEX_API_KEY` is accepted as exec authentication when `codex login status` reports logged out because that probe does not inspect environment based authentication. Unknown JSONL event types are retained and ignored unless they affect terminal correctness. Missing or changed required lifecycle events fail closed as protocol errors. Protocol fixtures cover every supported event type, unknown extensions, malformed JSON, failure events, missing terminal events, nonzero exits, signals, aborts, timeouts, resume, native review, cancellation, worker death, lock recovery, worktrees, and concurrent terminal writes. diff --git a/plugins/codex/scripts/codex-companion.mjs b/plugins/codex/scripts/codex-companion.mjs index e5e636a..eaaff64 100644 --- a/plugins/codex/scripts/codex-companion.mjs +++ b/plugins/codex/scripts/codex-companion.mjs @@ -88,7 +88,7 @@ const BACKGROUND_ABORT_CLAIM_WAIT_MS = 2000; const BACKGROUND_ABORT_CLEANUP_CONFIRM_MS = 1000; const BACKGROUND_ABORT_CLEANUP_POLL_MS = 50; const TESTED_VERSION_MIN = [0, 154, 0]; -const TESTED_VERSION_MAX = [0, 158, 0]; +const TESTED_VERSION_MAX = [0, 163, 0]; const CONTINUE_PROMPT = "Continue from the current Codex thread state. Complete the next highest value step and continue until the task is resolved."; const TRANSPORT_DIRECTORY_PREFIX = "codex-companion-input-"; const TRANSPORT_TOKEN_PATTERN = /^[a-f0-9]{48}$/; diff --git a/plugins/codex/scripts/lib/codex-exec.mjs b/plugins/codex/scripts/lib/codex-exec.mjs index 385f975..8ec1ce6 100644 --- a/plugins/codex/scripts/lib/codex-exec.mjs +++ b/plugins/codex/scripts/lib/codex-exec.mjs @@ -1237,10 +1237,14 @@ function classifiedFailureKind(message, fallback = "error") { if (/quota|insufficient (?:account )?(?:balance|credit|credits|funds)|balance (?:is )?exhausted|payment required|usage limit|billing|http(?:\/\d(?:\.\d)?)?\s+402|status(?: code)?\s*[:=]?\s*402/i.test(text)) { return "quota"; } - if (/rate[ _-]?limit|too many requests|http(?:\/\d(?:\.\d)?)?\s+429|status(?: code)?\s*[:=]?\s*429/i.test(text)) { + if (/rate[ _-]?limit|too many requests|http(?:\/\d(?:\.\d)?)?\s+429|status(?: code)?\s*[:=]?\s*429/i.test(text) || text.includes("Flex capacity unavailable.")) { return "rate_limited"; } - if (/misalignment policy|cyber policy|violat(?:ed|es|ion of) (?:the |our |an? )?[a-z ]{0,40}policy|content policy/i.test(text)) { + if (/misalignment policy|cyber policy|violat(?:ed|es|ion of) (?:the |our |an? )?[a-z ]{0,40}policy|content policy/i.test(text) || [ + "stream disconnected before completion: Incomplete response returned, reason: content_filter", + "This request has been flagged for possible cybersecurity risk.", + "This content was flagged for possible biological risk." + ].some((message) => text.includes(message))) { return "refused"; } if (/unauthori[sz]ed|unauthenticated|authentication|login required|invalid api key|api key/i.test(text)) { diff --git a/plugins/fusion/verified-versions.json b/plugins/fusion/verified-versions.json index 6a76fcb..f5cc36f 100644 --- a/plugins/fusion/verified-versions.json +++ b/plugins/fusion/verified-versions.json @@ -1,6 +1,6 @@ { - "codex-cli": "0.157.1", + "codex-cli": "0.162.1", "grok": "1.0.41", "cursor": "2026.10.01-e373342", - "verifiedAt": "2026-09-27" + "verifiedAt": "2026-10-13" } diff --git a/tests/breaker-check.test.mjs b/tests/breaker-check.test.mjs index e64c550..6e865b5 100644 --- a/tests/breaker-check.test.mjs +++ b/tests/breaker-check.test.mjs @@ -667,10 +667,10 @@ test("Codex version drift uses the registry verified version key", (t) => { writeRecord(jobFile(sandbox.codexState, "workspace", "auth"), { status: "error", failureKind: "auth", - codexVersion: "0.158.0", + codexVersion: "0.163.0", finishedAt: new Date(Date.now() - 2 * 60000).toISOString() }); - assert.match(run(sandbox).stdout, /; codex 0\.158\.0 installed, contract verified on 0\.157\.1/); + assert.match(run(sandbox).stdout, /; codex 0\.163\.0 installed, contract verified on 0\.162\.1/); }); test("a failure outside the lookback window is silent", (t) => { diff --git a/tests/codex-companion.test.mjs b/tests/codex-companion.test.mjs index 1b026a7..866e996 100644 --- a/tests/codex-companion.test.mjs +++ b/tests/codex-companion.test.mjs @@ -549,7 +549,7 @@ test("task stays foreground by default and persists the complete terminal record assert.equal(entry.record.resolvedModel, "gpt-test"); assert.equal(entry.record.resolvedEffort, "max"); assert.equal(entry.record.tokenUsageAvailability, "available"); - assert.equal(entry.record.codexVersion, "0.157.1"); + assert.equal(entry.record.codexVersion, "0.162.1"); assert.equal(fs.statSync(entry.file).mode & 0o777, 0o600); assert.equal(fs.statSync(path.dirname(entry.file)).mode & 0o777, 0o700); }); @@ -1259,7 +1259,7 @@ test("task rejects an outdated Codex CLI before execution", (t) => { }); test("task proceeds with tested, alpha hotfix, and newer Codex CLI versions", (t) => { - for (const version of ["0.154.0", "0.154.1", "0.155.0-alpha.5.1", "0.155.0", "0.156.0", "0.157.1", "0.158.0"]) { + for (const version of ["0.154.0", "0.154.1", "0.155.0-alpha.5.1", "0.155.0", "0.156.0", "0.157.1", "0.162.1", "0.163.0"]) { const sandbox = makeSandbox(t); const result = runCompanion(["task", "--json", "do work"], { cwd: sandbox.workDir, @@ -2403,7 +2403,7 @@ test("setup verifies authentication and the tested Codex version interval", (t) assert.equal(readyReport.ready, true); assert.equal(readyReport.compatibility, "tested"); assert.equal(readyReport.authenticated, true); - for (const version of ["0.154.1", "0.155.0-alpha.5.1", "0.155.0", "0.156.0", "0.157.1"]) { + for (const version of ["0.154.1", "0.155.0-alpha.5.1", "0.155.0", "0.156.0", "0.157.1", "0.162.1"]) { const alpha = runCompanion(["setup", "--json"], { cwd: sandbox.workDir, env: envFor(sandbox, { FAKE_CODEX_VERSION: version }) @@ -2421,7 +2421,7 @@ test("setup verifies authentication and the tested Codex version interval", (t) assert.doesNotMatch(JSON.stringify(JSON.parse(redacted.stdout)), /sk-secretvalue123/); const redactedVersion = runCompanion(["setup", "--json"], { cwd: sandbox.workDir, - env: envFor(sandbox, { FAKE_CODEX_VERSION_OUTPUT: "codex-cli 0.157.1 access_token=secretversionvalue" }) + env: envFor(sandbox, { FAKE_CODEX_VERSION_OUTPUT: "codex-cli 0.162.1 access_token=secretversionvalue" }) }); assert.equal(redactedVersion.status, 0, redactedVersion.stderr); assert.doesNotMatch(JSON.stringify(JSON.parse(redactedVersion.stdout)), /secretversionvalue/); @@ -2443,12 +2443,12 @@ test("setup verifies authentication and the tested Codex version interval", (t) assert.doesNotMatch(apiKey.stdout, /codex-test-key/); const unsupported = runCompanion(["setup", "--json"], { cwd: sandbox.workDir, - env: envFor(sandbox, { FAKE_CODEX_VERSION: "0.158.0" }) + env: envFor(sandbox, { FAKE_CODEX_VERSION: "0.163.0" }) }); assert.equal(unsupported.status, 1); const unsupportedReport = JSON.parse(unsupported.stdout); assert.equal(unsupportedReport.ready, false); - assert.match(unsupportedReport.compatibility, /newer than the tested interval \(0\.154\.0 to before 0\.158\.0\)/); + assert.match(unsupportedReport.compatibility, /newer than the tested interval \(0\.154\.0 to before 0\.163\.0\)/); assert.match(unsupportedReport.nextSteps.join("\n"), /A verification pass is advised\./); }); diff --git a/tests/codex-exec.test.mjs b/tests/codex-exec.test.mjs index 32173a6..3a6ed99 100644 --- a/tests/codex-exec.test.mjs +++ b/tests/codex-exec.test.mjs @@ -162,12 +162,12 @@ async function waitForProcessExit(pid, ownsProcessGroup = false) { test("version and availability probe the configured Codex binary", () => { const env = { ...process.env, CODEX_BIN: fakeCodex }; - assert.equal(getCodexVersion({ env }), "0.157.1"); + assert.equal(getCodexVersion({ env }), "0.162.1"); assert.deepEqual(getCodexAvailability({ env }), { available: true, bin: fakeCodex, - version: "0.157.1", - rawVersion: "codex-cli 0.157.1", + version: "0.162.1", + rawVersion: "codex-cli 0.162.1", exitCode: 0, errorMessage: null }); diff --git a/tests/failure-kinds.test.mjs b/tests/failure-kinds.test.mjs index 3cc130f..4e43f7a 100644 --- a/tests/failure-kinds.test.mjs +++ b/tests/failure-kinds.test.mjs @@ -97,6 +97,29 @@ test("a Codex refusal turn failure yields failure kind refused", (t) => { assert.match(resultOutput.stdout, /^failure: refused$/m); }); +for (const [message, failureKind] of [ + ["Flex capacity unavailable.", "rate_limited"], + ["stream disconnected before completion: Incomplete response returned, reason: content_filter", "refused"], + ["This request has been flagged for possible cybersecurity risk.", "refused"], + ["This content was flagged for possible biological risk.", "refused"], + ["The local disk has capacity for this request.", "error"] +]) { + test(`a Codex turn failure saying ${JSON.stringify(message)} yields failure kind ${failureKind}`, (t) => { + const sandbox = makeCodexSandbox(t); + const env = codexEnvFor(sandbox, { + FAKE_CODEX_MODE: "failed", + FAKE_CODEX_TURN_FAILURE_MESSAGE: message + }); + const result = runCodexCompanion(["task", "doomed"], { cwd: sandbox.workDir, env }); + assert.notStrictEqual(result.status, 0); + assert.match(result.stdout, new RegExp(`^failure: ${failureKind}$`, "m")); + const [record] = codexJobRecords(sandbox); + assert.strictEqual(record.status, "error"); + assert.strictEqual(record.failureKind, failureKind); + assert.strictEqual(record.errorMessage, message); + }); +} + test("auth stderr yields failure kind auth", (t) => { const sandbox = makeSandbox(t); const env = envFor(sandbox, { FAKE_GROK_MODE: "auth-error" }); diff --git a/tests/fake-codex b/tests/fake-codex index 448f483..725e4e1 100755 --- a/tests/fake-codex +++ b/tests/fake-codex @@ -65,7 +65,7 @@ if (UNMODELLED_SUBCOMMANDS.has(args[0])) { } if (args.includes("--version")) { - process.stdout.write(`${process.env.FAKE_CODEX_VERSION_OUTPUT || `codex-cli ${process.env.FAKE_CODEX_VERSION || "0.157.1"}`}\n`); + process.stdout.write(`${process.env.FAKE_CODEX_VERSION_OUTPUT || `codex-cli ${process.env.FAKE_CODEX_VERSION || "0.162.1"}`}\n`); process.exit(0); } diff --git a/tests/verified-versions.test.mjs b/tests/verified-versions.test.mjs index 5ab7e00..858a14b 100644 --- a/tests/verified-versions.test.mjs +++ b/tests/verified-versions.test.mjs @@ -16,7 +16,7 @@ test("verified peer CLI versions are complete and well formed", () => { verifiedVersions = readVerifiedVersions(); }); assert.deepEqual(Object.keys(verifiedVersions).sort(), ["codex-cli", "cursor", "grok", "verifiedAt"]); - assert.deepEqual(verifiedVersions, { "codex-cli": "0.157.1", grok: "1.0.41", cursor: "2026.10.01-e373342", verifiedAt: "2026-09-27" }); + assert.deepEqual(verifiedVersions, { "codex-cli": "0.162.1", grok: "1.0.41", cursor: "2026.10.01-e373342", verifiedAt: "2026-10-13" }); assert.match(verifiedVersions["codex-cli"], /^\d+\.\d+\.\d+$/); assert.match(verifiedVersions.grok, /^\d+\.\d+\.\d+$/); assert.match(verifiedVersions.cursor, /^\d{4}\.\d{2}\.\d{2}-[a-f0-9]+$/); From 26da8446afde72a0e7e66d4901cf8fa1006acf42 Mon Sep 17 00:00:00 2001 From: Harry Yep Date: Tue, 13 Oct 2026 17:25:49 +0800 Subject: [PATCH 2/8] feat(grok): pin turn summary and file acceleration off and drop the 1.0.52 env surfaces --- plugins/grok/scripts/grok-companion.mjs | 4 +- plugins/grok/scripts/lib/grok-exec.mjs | 7 ++ tests/fake-grok | 7 ++ tests/grok-auth-refresh.test.mjs | 26 +++- tests/grok-models.test.mjs | 14 ++- tests/grok-upstream-contract.test.mjs | 150 +++++++++++++----------- 6 files changed, 139 insertions(+), 69 deletions(-) diff --git a/plugins/grok/scripts/grok-companion.mjs b/plugins/grok/scripts/grok-companion.mjs index 75c8b80..09569a1 100644 --- a/plugins/grok/scripts/grok-companion.mjs +++ b/plugins/grok/scripts/grok-companion.mjs @@ -287,10 +287,12 @@ function modelsChildEnv() { delete env[key]; } } - for (const key of ["GROK_CLAUDE_MARKER_OVERRIDE", "_GROK_CLAUDE_MARKER_OVERRIDE", "GROK_TEST_VERSION", "GROK_SANDBOX", "GROK_LEADER_SOCKET", "GROK_SUBAGENTS_MAX_DEPTH", "GROK_WEB_FETCH_PROXY", "GROK_WEB_FETCH_ALLOW_LOCAL"]) { + for (const key of ["GROK_CLAUDE_MARKER_OVERRIDE", "_GROK_CLAUDE_MARKER_OVERRIDE", "GROK_TEST_VERSION", "GROK_SANDBOX", "GROK_LEADER_SOCKET", "GROK_FILE_ACCELERATION_ROUTES", "GROK_SANDBOX_MODE", "GROK_ARTIFACTS_BIND_REMOUNT", "GROK_CURSOR_WORKER_API_BASE", "GROK_CURSOR_WORKER_COMMAND", "GROK_SUBAGENTS_MAX_DEPTH", "GROK_WEB_FETCH_PROXY", "GROK_WEB_FETCH_ALLOW_LOCAL"]) { delete env[key]; } env.GROK_AUTO_WAKE = "false"; + env.GROK_TURN_SUMMARY = "false"; + env.GROK_FILE_ACCELERATION = "false"; env.RUST_LOG = "xai_grok_agent::builder=debug,xai_grok_sandbox=warn,xai_grok_shell::session::acp_session::turn=warn"; env.GROK_DISABLE_AUTOUPDATER = "1"; env.GROK_MANAGED_MCPS_ENABLED = "false"; diff --git a/plugins/grok/scripts/lib/grok-exec.mjs b/plugins/grok/scripts/lib/grok-exec.mjs index b37563b..81fe0da 100644 --- a/plugins/grok/scripts/lib/grok-exec.mjs +++ b/plugins/grok/scripts/lib/grok-exec.mjs @@ -220,7 +220,14 @@ function buildGrokChildEnv(env = process.env, { memory = false, web = false } = delete childEnv.GROK_TEST_VERSION; delete childEnv.GROK_SANDBOX; delete childEnv.GROK_LEADER_SOCKET; + delete childEnv.GROK_FILE_ACCELERATION_ROUTES; + delete childEnv.GROK_SANDBOX_MODE; + delete childEnv.GROK_ARTIFACTS_BIND_REMOUNT; + delete childEnv.GROK_CURSOR_WORKER_API_BASE; + delete childEnv.GROK_CURSOR_WORKER_COMMAND; childEnv.GROK_AUTO_WAKE = "false"; + childEnv.GROK_TURN_SUMMARY = "false"; + childEnv.GROK_FILE_ACCELERATION = "false"; delete childEnv.GROK_SUBAGENTS_MAX_DEPTH; delete childEnv.GROK_WEB_FETCH_PROXY; delete childEnv.GROK_WEB_FETCH_ALLOW_LOCAL; diff --git a/tests/fake-grok b/tests/fake-grok index 9c1325f..53d9b64 100755 --- a/tests/fake-grok +++ b/tests/fake-grok @@ -90,6 +90,13 @@ key === "GROK_SANDBOX" || key === "GROK_LEADER_SOCKET" || key === "GROK_AUTO_WAKE" || + key === "GROK_TURN_SUMMARY" || + key === "GROK_FILE_ACCELERATION" || + key === "GROK_FILE_ACCELERATION_ROUTES" || + key === "GROK_SANDBOX_MODE" || + key === "GROK_ARTIFACTS_BIND_REMOUNT" || + key === "GROK_CURSOR_WORKER_API_BASE" || + key === "GROK_CURSOR_WORKER_COMMAND" || key === "GROK_SUBAGENTS_MAX_DEPTH" || key === "GROK_WEB_FETCH_PROXY" || key === "GROK_WEB_FETCH_ALLOW_LOCAL" || diff --git a/tests/grok-auth-refresh.test.mjs b/tests/grok-auth-refresh.test.mjs index efe16a6..6b14eff 100644 --- a/tests/grok-auth-refresh.test.mjs +++ b/tests/grok-auth-refresh.test.mjs @@ -42,7 +42,14 @@ test("direct managed execution records a stale pre-run refresh", async (t) => { FAKE_GROK_MODELS_TOUCH: file, FAKE_GROK_MODELS_ENV_FILE: modelsEnvFile, ANTHROPIC_API_KEY: "must-be-scrubbed", - XAI_API_KEY: "retained-for-auth" + XAI_API_KEY: "retained-for-auth", + GROK_TURN_SUMMARY: "true", + GROK_FILE_ACCELERATION: "true", + GROK_FILE_ACCELERATION_ROUTES: "inherited-routes", + GROK_SANDBOX_MODE: "inherited-mode", + GROK_ARTIFACTS_BIND_REMOUNT: "inherited-remount", + GROK_CURSOR_WORKER_API_BASE: "https://worker.example.test", + GROK_CURSOR_WORKER_COMMAND: "inherited-worker" }))); assert.equal(result.exitCode, 0); assert.equal(result.authRefresh.reason, "stale"); @@ -52,6 +59,11 @@ test("direct managed execution records a stale pre-run refresh", async (t) => { const childEnv = JSON.parse(fs.readFileSync(modelsEnvFile, "utf8")); assert.equal(childEnv.ANTHROPIC_API_KEY, undefined); assert.equal(childEnv.XAI_API_KEY, "retained-for-auth"); + assert.equal(childEnv.GROK_TURN_SUMMARY, "false"); + assert.equal(childEnv.GROK_FILE_ACCELERATION, "false"); + for (const key of ["GROK_FILE_ACCELERATION_ROUTES", "GROK_SANDBOX_MODE", "GROK_ARTIFACTS_BIND_REMOUNT", "GROK_CURSOR_WORKER_API_BASE", "GROK_CURSOR_WORKER_COMMAND"]) { + assert.equal(childEnv[key], undefined, key); + } }); test("direct managed execution retries after auth refresh", async (t) => { @@ -144,6 +156,13 @@ test("stale auth runs one unsandboxed models refresh with the scrubbed child env ANTHROPIC_API_KEY: "must-be-scrubbed", XAI_API_KEY: "retained-for-auth", GROK_MANAGED_MCPS_ENABLED: "true", + GROK_TURN_SUMMARY: "true", + GROK_FILE_ACCELERATION: "true", + GROK_FILE_ACCELERATION_ROUTES: "inherited-routes", + GROK_SANDBOX_MODE: "inherited-mode", + GROK_ARTIFACTS_BIND_REMOUNT: "inherited-remount", + GROK_CURSOR_WORKER_API_BASE: "https://worker.example.test", + GROK_CURSOR_WORKER_COMMAND: "inherited-worker", CLAUDE_CODE_SESSION_ID: "must-be-scrubbed" }); const result = runCompanion(["task", "inspect the repository"], { cwd: sandbox.workDir, env }); @@ -155,6 +174,11 @@ test("stale auth runs one unsandboxed models refresh with the scrubbed child env assert.equal(childEnv.CLAUDE_CODE_SESSION_ID, undefined); assert.equal(childEnv.XAI_API_KEY, "retained-for-auth"); assert.equal(childEnv.GROK_MANAGED_MCPS_ENABLED, "false"); + assert.equal(childEnv.GROK_TURN_SUMMARY, "false"); + assert.equal(childEnv.GROK_FILE_ACCELERATION, "false"); + for (const key of ["GROK_FILE_ACCELERATION_ROUTES", "GROK_SANDBOX_MODE", "GROK_ARTIFACTS_BIND_REMOUNT", "GROK_CURSOR_WORKER_API_BASE", "GROK_CURSOR_WORKER_COMMAND"]) { + assert.equal(childEnv[key], undefined, key); + } const [record] = jobRecords(sandbox.dataDir); assert.equal(record.authRefresh.reason, "stale"); assert.equal(record.authRefresh.refreshed, true); diff --git a/tests/grok-models.test.mjs b/tests/grok-models.test.mjs index ff0e728..1dc2b35 100644 --- a/tests/grok-models.test.mjs +++ b/tests/grok-models.test.mjs @@ -41,7 +41,14 @@ test("models uses the scrubbed Grok child environment", (t) => { FAKE_GROK_MODELS_ENV_FILE: envFile, ANTHROPIC_API_KEY: "must-be-scrubbed", XAI_API_KEY: "retained-for-auth", - CLAUDE_PLUGIN_ROOT: "must-be-scrubbed" + CLAUDE_PLUGIN_ROOT: "must-be-scrubbed", + GROK_TURN_SUMMARY: "true", + GROK_FILE_ACCELERATION: "true", + GROK_FILE_ACCELERATION_ROUTES: "inherited-routes", + GROK_SANDBOX_MODE: "inherited-mode", + GROK_ARTIFACTS_BIND_REMOUNT: "inherited-remount", + GROK_CURSOR_WORKER_API_BASE: "https://worker.example.test", + GROK_CURSOR_WORKER_COMMAND: "inherited-worker" }) }); assert.equal(result.status, 0, result.stderr); const childEnv = JSON.parse(fs.readFileSync(envFile, "utf8")); @@ -50,6 +57,11 @@ test("models uses the scrubbed Grok child environment", (t) => { assert.equal(childEnv.XAI_API_KEY, "retained-for-auth"); assert.equal(childEnv.GROK_DISABLE_AUTOUPDATER, "1"); assert.equal(childEnv.GROK_MEMORY, "0"); + assert.equal(childEnv.GROK_TURN_SUMMARY, "false"); + assert.equal(childEnv.GROK_FILE_ACCELERATION, "false"); + for (const key of ["GROK_FILE_ACCELERATION_ROUTES", "GROK_SANDBOX_MODE", "GROK_ARTIFACTS_BIND_REMOUNT", "GROK_CURSOR_WORKER_API_BASE", "GROK_CURSOR_WORKER_COMMAND"]) { + assert.equal(childEnv[key], undefined, key); + } }); test("models classifies CLI, auth, setup, and listing failures", (t) => { diff --git a/tests/grok-upstream-contract.test.mjs b/tests/grok-upstream-contract.test.mjs index 1eed7dc..6721a14 100644 --- a/tests/grok-upstream-contract.test.mjs +++ b/tests/grok-upstream-contract.test.mjs @@ -7,6 +7,7 @@ import path from "node:path"; import test from "node:test"; import { getProcessIdentity } from "../plugins/grok/scripts/lib/process-identity.mjs"; +import { initFixtureRepo } from "./lib/git-fixture.mjs"; import { envFor, @@ -515,73 +516,90 @@ test("setup reports grok doctor as unavailable when the subcommand is missing", assert.match(rendered.stdout, /grok doctor: not available/); }); -test("Grok child environment removes Claude integration state and disables implicit context bridges", async (t) => { - const sandbox = makeSandbox(t); - const envFile = path.join(sandbox.root, "child-env.json"); - const enabledBridges = Object.fromEntries(bridgeEnvironmentKeys.map((key) => [key, "true"])); - const env = envFor(sandbox, { - ...enabledBridges, - CLAUDE_CODE_SESSION_ID: "claude-session-secret", - CLAUDE_CODE_PRIVATE_TOKEN: "claude-code-secret", - CLAUDE_PLUGIN_ROOT: "/private/plugin/root", - CLAUDE_PLUGIN_PRIVATE_TOKEN: "claude-plugin-secret", - ANTHROPIC_API_KEY: "anthropic-secret", - CODEX_API_KEY: "codex-secret", - OPENAI_API_KEY: "openai-secret", - XAI_API_KEY: "xai-test-auth", - DATABASE_URL: "postgres://user:password@db.example.test/app", - REDIS_URL: "redis://user:password@redis.example.test/0", - SENTRY_DSN: "https://public:secret@sentry.example.test/1", - SSH_AUTH_SOCK: "/private/ssh-agent.sock", - GROK_CLAUDE_MARKER_OVERRIDE: "marker-secret", - _GROK_CLAUDE_MARKER_OVERRIDE: "private-marker-secret", - GROK_MEMORY: "1", - GROK_SUBAGENTS: "1", - GROK_MANAGED_MCPS_ENABLED: "true", - GROK_MANAGED_MCP_GATEWAY_TOOLS_ENABLED: "true", - GROK_WEB_FETCH: "1", - GROK_TEST_VERSION: "9.9.9", - GROK_SANDBOX: "inherited-sandbox", - GROK_LEADER_SOCKET: "/tmp/inherited-leader.sock", - GROK_AUTO_WAKE: "1", - GROK_SUBAGENTS_MAX_DEPTH: "99", - GROK_WEB_FETCH_PROXY: "http://proxy.example.test", - GROK_WEB_FETCH_ALLOW_LOCAL: "1", - GROK_COMPANION_CONTINUITY_POLICY: "claude-session", - RUST_LOG: "off", - FAKE_GROK_ENV_FILE: envFile, - FAKE_GROK_MODE: "ok", +for (const mode of ["consult", "write", "review"]) { + test(`${mode} child environment removes inherited Grok controls and disables implicit context bridges`, async (t) => { + const sandbox = makeSandbox(t); + if (mode === "review") { + initFixtureRepo(sandbox.workDir); + } + const enabledBridges = Object.fromEntries(bridgeEnvironmentKeys.map((key) => [key, "true"])); + const parentEnv = { + ...enabledBridges, + CLAUDE_CODE_SESSION_ID: "claude-session-secret", + CLAUDE_CODE_PRIVATE_TOKEN: "claude-code-secret", + CLAUDE_PLUGIN_ROOT: "/private/plugin/root", + CLAUDE_PLUGIN_PRIVATE_TOKEN: "claude-plugin-secret", + ANTHROPIC_API_KEY: "anthropic-secret", + CODEX_API_KEY: "codex-secret", + OPENAI_API_KEY: "openai-secret", + XAI_API_KEY: "xai-test-auth", + DATABASE_URL: "postgres://user:password@db.example.test/app", + REDIS_URL: "redis://user:password@redis.example.test/0", + SENTRY_DSN: "https://public:secret@sentry.example.test/1", + SSH_AUTH_SOCK: "/private/ssh-agent.sock", + GROK_CLAUDE_MARKER_OVERRIDE: "marker-secret", + _GROK_CLAUDE_MARKER_OVERRIDE: "private-marker-secret", + GROK_MEMORY: "1", + GROK_SUBAGENTS: "1", + GROK_MANAGED_MCPS_ENABLED: "true", + GROK_MANAGED_MCP_GATEWAY_TOOLS_ENABLED: "true", + GROK_WEB_FETCH: "1", + GROK_TEST_VERSION: "9.9.9", + GROK_SANDBOX: "inherited-sandbox", + GROK_LEADER_SOCKET: "/tmp/inherited-leader.sock", + GROK_AUTO_WAKE: "1", + GROK_TURN_SUMMARY: "true", + GROK_FILE_ACCELERATION: "true", + GROK_FILE_ACCELERATION_ROUTES: "inherited-routes", + GROK_SANDBOX_MODE: "inherited-mode", + GROK_ARTIFACTS_BIND_REMOUNT: "inherited-remount", + GROK_CURSOR_WORKER_API_BASE: "https://worker.example.test", + GROK_CURSOR_WORKER_COMMAND: "inherited-worker", + GROK_SUBAGENTS_MAX_DEPTH: "99", + GROK_WEB_FETCH_PROXY: "http://proxy.example.test", + GROK_WEB_FETCH_ALLOW_LOCAL: "1", + GROK_COMPANION_CONTINUITY_POLICY: "claude-session", + RUST_LOG: "off", + }; + const envFile = path.join(sandbox.root, `${mode}-child-env.json`); + const env = envFor(sandbox, { ...parentEnv, FAKE_GROK_ENV_FILE: envFile, FAKE_GROK_MODE: mode === "review" ? "review-ok" : "ok" }); + if (mode === "review") { + const result = runCompanion(["review", "--json"], { cwd: sandbox.workDir, env }); + assert.equal(result.status, 0, result.stderr); + } else { + const result = await runGrok(directOptions(sandbox, "ok", { mode, env })); + assert.equal(result.exitCode, 0, result.errorMessage); + } + const childEnv = JSON.parse(fs.readFileSync(envFile, "utf8")); + assert.ok(!Object.keys(childEnv).some((key) => key.startsWith("CLAUDE_CODE_"))); + assert.ok(!Object.keys(childEnv).some((key) => key.startsWith("CLAUDE_PLUGIN_"))); + for (const key of ["ANTHROPIC_API_KEY", "CODEX_API_KEY", "OPENAI_API_KEY", "DATABASE_URL", "REDIS_URL", "SENTRY_DSN", "SSH_AUTH_SOCK", "GROK_CLAUDE_MARKER_OVERRIDE", "_GROK_CLAUDE_MARKER_OVERRIDE"]) { + assert.equal(childEnv[key], undefined, key); + } + assert.equal(childEnv.XAI_API_KEY, "xai-test-auth"); + assert.equal(childEnv.GROK_DISABLE_AUTOUPDATER, "1"); + assert.equal(childEnv.GROK_MANAGED_MCPS_ENABLED, "false"); + assert.equal(childEnv.GROK_MANAGED_MCP_GATEWAY_TOOLS_ENABLED, "false"); + assert.equal(childEnv.GROK_WEB_FETCH, "0"); + assert.equal(childEnv.GROK_MEMORY, "0"); + assert.equal(childEnv.GROK_SUBAGENTS, "0"); + assert.equal(childEnv.GROK_AUTO_WAKE, "false"); + assert.equal(childEnv.GROK_TURN_SUMMARY, "false"); + assert.equal(childEnv.GROK_FILE_ACCELERATION, "false"); + for (const key of ["GROK_TEST_VERSION", "GROK_SANDBOX", "GROK_LEADER_SOCKET", "GROK_FILE_ACCELERATION_ROUTES", "GROK_SANDBOX_MODE", "GROK_ARTIFACTS_BIND_REMOUNT", "GROK_CURSOR_WORKER_API_BASE", "GROK_CURSOR_WORKER_COMMAND", "GROK_SUBAGENTS_MAX_DEPTH", "GROK_WEB_FETCH_PROXY", "GROK_WEB_FETCH_ALLOW_LOCAL"]) { + assert.equal(childEnv[key], undefined, key); + } + assert.equal(childEnv.GROK_COMPANION_CONTINUITY_POLICY, undefined); + assert.equal(childEnv.RUST_LOG, "xai_grok_agent::builder=debug,xai_grok_sandbox=warn,xai_grok_shell::session::acp_session::turn=warn"); + assert.equal(path.isAbsolute(childEnv.TMPDIR), true); + assert.equal(path.dirname(childEnv.TMPDIR), fs.realpathSync(mode === "review" ? env.TMPDIR : os.tmpdir())); + assert.match(path.basename(childEnv.TMPDIR), /^grok-companion-run-/); + assert.equal(fs.existsSync(childEnv.TMPDIR), false); + for (const key of bridgeEnvironmentKeys) { + assert.equal(childEnv[key], "false", key); + } }); - const result = await runGrok(directOptions(sandbox, "ok", { env })); - - assert.equal(result.exitCode, 0); - const childEnv = JSON.parse(fs.readFileSync(envFile, "utf8")); - assert.ok(!Object.keys(childEnv).some((key) => key.startsWith("CLAUDE_CODE_"))); - assert.ok(!Object.keys(childEnv).some((key) => key.startsWith("CLAUDE_PLUGIN_"))); - for (const key of ["ANTHROPIC_API_KEY", "CODEX_API_KEY", "OPENAI_API_KEY", "DATABASE_URL", "REDIS_URL", "SENTRY_DSN", "SSH_AUTH_SOCK", "GROK_CLAUDE_MARKER_OVERRIDE", "_GROK_CLAUDE_MARKER_OVERRIDE"]) { - assert.equal(childEnv[key], undefined, key); - } - assert.equal(childEnv.XAI_API_KEY, "xai-test-auth"); - assert.equal(childEnv.GROK_DISABLE_AUTOUPDATER, "1"); - assert.equal(childEnv.GROK_MANAGED_MCPS_ENABLED, "false"); - assert.equal(childEnv.GROK_MANAGED_MCP_GATEWAY_TOOLS_ENABLED, "false"); - assert.equal(childEnv.GROK_WEB_FETCH, "0"); - assert.equal(childEnv.GROK_MEMORY, "0"); - assert.equal(childEnv.GROK_SUBAGENTS, "0"); - assert.equal(childEnv.GROK_AUTO_WAKE, "false"); - for (const key of ["GROK_TEST_VERSION", "GROK_SANDBOX", "GROK_LEADER_SOCKET", "GROK_SUBAGENTS_MAX_DEPTH", "GROK_WEB_FETCH_PROXY", "GROK_WEB_FETCH_ALLOW_LOCAL"]) { - assert.equal(childEnv[key], undefined, key); - } - assert.equal(childEnv.GROK_COMPANION_CONTINUITY_POLICY, undefined); - assert.equal(childEnv.RUST_LOG, "xai_grok_agent::builder=debug,xai_grok_sandbox=warn,xai_grok_shell::session::acp_session::turn=warn"); - assert.equal(path.isAbsolute(childEnv.TMPDIR), true); - assert.equal(path.dirname(childEnv.TMPDIR), fs.realpathSync(os.tmpdir())); - assert.match(path.basename(childEnv.TMPDIR), /^grok-companion-run-/); - assert.equal(fs.existsSync(childEnv.TMPDIR), false); - for (const key of bridgeEnvironmentKeys) { - assert.equal(childEnv[key], "false", key); - } -}); +} test("web-enabled managed runs enable Grok web fetch for the child", async (t) => { const sandbox = makeSandbox(t); From cbb3c4f46d7dd1f11e40b75940eee41339f1e807 Mon Sep 17 00:00:00 2001 From: Harry Yep Date: Tue, 13 Oct 2026 17:25:49 +0800 Subject: [PATCH 3/8] feat(bench): add the lane comparison instrument and blind consult briefs --- bench/METHODOLOGY.md | 12 + bench/lanes/consult/blind-review.md | 7 + bench/lanes/consult/clean-review.md | 7 + bench/lanes/consult/emberflow-research.md | 15 + bench/lanes/consult/lru-recency.md | 3 + bench/lanes/run.mjs | 312 ++++++++++++++++++ bench/lanes/summarize.mjs | 90 +++++ bench/lib.mjs | 79 +++++ bench/run.mjs | 80 +---- tests/bench-lanes.test.mjs | 203 ++++++++++++ tests/fixtures/bench-lanes/fake-companion.mjs | 48 +++ .../bench-lanes/tasks/T01-test/brief.md | 1 + .../tasks/T01-test/fixtures/input.txt | 1 + .../bench-lanes/tasks/T01-test/verify.sh | 6 + 14 files changed, 786 insertions(+), 78 deletions(-) create mode 100644 bench/lanes/consult/blind-review.md create mode 100644 bench/lanes/consult/clean-review.md create mode 100644 bench/lanes/consult/emberflow-research.md create mode 100644 bench/lanes/consult/lru-recency.md create mode 100644 bench/lanes/run.mjs create mode 100644 bench/lanes/summarize.mjs create mode 100644 bench/lib.mjs create mode 100644 tests/bench-lanes.test.mjs create mode 100644 tests/fixtures/bench-lanes/fake-companion.mjs create mode 100644 tests/fixtures/bench-lanes/tasks/T01-test/brief.md create mode 100644 tests/fixtures/bench-lanes/tasks/T01-test/fixtures/input.txt create mode 100644 tests/fixtures/bench-lanes/tasks/T01-test/verify.sh diff --git a/bench/METHODOLOGY.md b/bench/METHODOLOGY.md index 3dd8e32..0a1207d 100644 --- a/bench/METHODOLOGY.md +++ b/bench/METHODOLOGY.md @@ -85,3 +85,15 @@ Before any results are published, the following must exist in the repo: the runn ## Classifier calibration `bench/jev/calibrate.mjs` is a separate instrument from the task suite and makes no claim under test. It guards the brief readiness check: the 0.7 flag threshold in `plugins/fusion/scripts/brief-sensor.mjs` is only meaningful for the pinned Jev model, so a model bump is calibrated before the pin in `plugins/fusion/scripts/lib/jev.mjs` moves. The script sends the twelve labeled briefs in `bench/jev/briefs.json` (two per flaw plus four clean controls) with the sensor's own questions and threshold, and exits 1 when a flawed brief is not flagged on its labeled check, a clean brief is flagged on any check, or a call returns no answer; exit 2 means `TYPESAFE_API_KEY` is not set. `--model ` scores a candidate instead of the pinned model. `--retained` additionally scores this machine's accepted peer briefs for the current repository only, skipping review prompts, and reports how many were flagged as a false positive proxy without affecting the exit code; it sends those briefs to TypeSafe AI, so it is opt in and never reads another repository's job state. + +## Lane comparison + +`bench/lanes/run.mjs` is a second instrument outside the claims under test. It measures one routing lane at a time, so the per user capability scores behind `/fusion:config` can rest on matched evidence rather than vendor benchmarks alone. One invocation runs one attempt: it copies a task's fixtures into a fresh Git repository, sends one brief through one engine companion with an explicit `--model` and `--effort`, and appends one record to `runs.jsonl` in the results directory, next to an `env.json` written once per directory with the same environment pinning as the runner. It resolves each companion the same way Fusion does, so the instrument measures the installed lane rather than a test double. + +- Write mode sends the task's `brief.md` with `--write` and then runs the task's `verify.sh` against the workspace from outside it, so the verdict is pass or fail exactly as in the task suite. The staged diff is saved next to the record for audit and stays out of the tree. +- Consult mode sends a read only brief from `bench/lanes/consult/` against a task's fixtures (or another directory inside the task, such as `solution`) and saves the companion's answer next to the record with the verdict `unscored`. The orchestrator scores it afterwards against the task's own solution or answer key, which the workspace never contains. +- An engine failure that left a job record, quota exhaustion included, counts as a failed attempt with its failure kind recorded, never as an exclusion. `infra_failure` is reserved for attempts the instrument could not measure: the companion would not start, timed out, printed no job id, or left no job record. + +`bench/lanes/summarize.mjs` groups records by task, engine, model, and effort and reports attempts, verdicts, median wall clock, median input and output tokens, and failure kinds. Consult answers are scored on four measures: correct findings that match the key, false claims that do not reproduce against the documented behavior, evidence quality (whether each claim cites the right file and line), and adjudication burden (how many claims the orchestrator had to settle by hand). The four consult briefs pair with task fixtures as follows: `lru-recency.md` diagnoses the T01 seeded bug from its symptom report, `blind-review.md` reviews the three T06 packages against their documented behavior with no symptoms given, `emberflow-research.md` answers the T04 research question in the reply instead of a file, and `clean-review.md` reviews the already correct T02 solution, so every reported defect is a false claim. + +A lane snapshot lives under `bench/results/lanes-/` with `runs.jsonl` passed through `bench/redact.mjs`, `env.json`, and the rendered summary with its consult scores. Like the task suite snapshots, it is descriptive: three repetitions per lane and task at most, one machine, one account per engine, and model serving that drifts within days, so it supports a routing decision for the snapshot date and nothing broader. diff --git a/bench/lanes/consult/blind-review.md b/bench/lanes/consult/blind-review.md new file mode 100644 index 0000000..e4d30d2 --- /dev/null +++ b/bench/lanes/consult/blind-review.md @@ -0,0 +1,7 @@ +This workspace holds three independent Node ESM packages. Their documented behavior: + +- `csv-parser/index.js` exports `parseCsvLine(line)`, which splits one CSV line into fields. Fields are separated by commas, a field wrapped in double quotes may contain commas, a doubled double quote inside a quoted field stands for one literal double quote, and surrounding whitespace is trimmed from each field. +- `duration-formatter/index.js` exports `formatDuration(totalSeconds)`, which truncates its input to whole seconds, treats negative input as 0, and formats the result as `m s` using completed minutes and the remaining seconds, or as `s` under one minute. +- `path-matcher/index.js` exports `matchesPath(pattern, candidate)`, which matches a slash separated path against a pattern in which `*` matches any run of characters within a single path segment and every other character matches itself literally. + +Review the three packages for correctness defects against this documented behavior. This is a read only consult: do not edit or create any file. Report every defect you can demonstrate, each with its file and line, a concrete call, the expected and actual results, and the minimal fix. Do not report style preferences or behavior the documentation leaves open. If a package has no defect, say so. Keep the report under 80 lines. diff --git a/bench/lanes/consult/clean-review.md b/bench/lanes/consult/clean-review.md new file mode 100644 index 0000000..7103ec7 --- /dev/null +++ b/bench/lanes/consult/clean-review.md @@ -0,0 +1,7 @@ +The library in this workspace is a numeric range helper exported from range.js. Its documented behavior: + +- `clamp(value, min, max)` throws a RangeError when min is greater than max, returns min when value is below min, returns max when value is above max, and otherwise returns value unchanged. +- `normalize(value, min, max)` throws a RangeError when max equals min, and otherwise maps the clamped value onto the 0 to 1 interval, so a value at min maps to 0 and a value at max maps to 1. +- `lerp(t, min, max)` returns the point a fraction t of the way from min to max, without clamping t. + +Review range.js for correctness defects against this documented behavior. This is a read only consult: do not edit or create any file. Report every defect you can demonstrate, each with its file and line, a concrete call, the expected and actual results, and the minimal fix. Do not report style preferences or behavior the documentation leaves open. If you find no defect, say so. Keep the report under 40 lines. diff --git a/bench/lanes/consult/emberflow-research.md b/bench/lanes/consult/emberflow-research.md new file mode 100644 index 0000000..29746ca --- /dev/null +++ b/bench/lanes/consult/emberflow-research.md @@ -0,0 +1,15 @@ +The corpus folder in this workspace holds documentation for Emberflow, a fictional workflow automation engine that reads its settings from an emberflow.yml file. The documents mix full changelogs, standalone release notes for individual versions, a configuration reference, a migration guide, a deprecation notice, a frequently asked questions page, a known issues list, and a support ticket digest, together covering the project's entire release history from its first stable version through its current version. Several of these documents state a configuration default that was accurate when that document was written but was changed again in a later release, so finding a value for a key in one document does not mean that value is still current; read enough of the corpus to confirm which value for each key the current version actually ships, not merely a value the key has held at some point in its history. + +Answer the following question about three settings read from emberflow.yml: queue.concurrency, retry.backoff_ms, and webhook.timeout_ms. For each of the three keys, determine the value it currently defaults to in the latest version covered by this corpus, and the version in which that current value was most recently set. A key's current value was most recently set in the latest version whose changelog entry changes that key, even when later versions have shipped since without touching it; for example, if a key was set to 10 in version 1.0.0 and no later version ever changes it again, then 10 from version 1.0.0 remains both its current value and the version of its most recent change, no matter how many later versions have shipped since. + +This is a read only consult: do not edit or create any file. Reply with your answer as a single fenced json block in exactly the shape below, with exactly these three top level keys spelled exactly as shown, each holding an object with exactly two fields: final_default as a JSON number, not a string, and changed_in_version as a JSON string holding a plain three part version number with no leading v, matching the version headings used in this corpus's changelogs, for example "4.0.0". After the block, cite the document and heading that settles each value. + +```json +{ + "queue_concurrency": { "final_default": 0, "changed_in_version": "0.0.0" }, + "retry_backoff_ms": { "final_default": 0, "changed_in_version": "0.0.0" }, + "webhook_timeout_ms": { "final_default": 0, "changed_in_version": "0.0.0" } +} +``` + +The zeroes above are placeholders that show the required shape only; replace every final_default and every changed_in_version with the value you determine from the corpus, and do not otherwise change the shape. diff --git a/bench/lanes/consult/lru-recency.md b/bench/lanes/consult/lru-recency.md new file mode 100644 index 0000000..8061456 --- /dev/null +++ b/bench/lanes/consult/lru-recency.md @@ -0,0 +1,3 @@ +The library in this workspace is a TTL aware LRU cache exported from index.js. A user reports that capacity eviction does not follow least recently used order. With maxSize 2, if they set a to 1, set b to 2, get a, then set c to 3, they expect has(a) and get(a) to still work and b to be gone, but a disappears and b stays. With maxSize 3, touching one key several times before adding a new key evicts the hot key instead of a cold one. + +Find the root cause. This is a read only consult: do not edit or create any file. Report every defect you can demonstrate, each with its file and line, the concrete calls that show it, the expected and actual results, and the minimal fix. If you find no defect, say so. Keep the report under 60 lines. diff --git a/bench/lanes/run.mjs b/bench/lanes/run.mjs new file mode 100644 index 0000000..60abd7a --- /dev/null +++ b/bench/lanes/run.mjs @@ -0,0 +1,312 @@ +#!/usr/bin/env node + +import { randomUUID } from "node:crypto"; +import { spawn, spawnSync } from "node:child_process"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { cliVersion, collectPluginVersions, readJsonFile } from "../lib.mjs"; +import { buildManifest } from "../manifest.mjs"; +import { newestEngineCompanion } from "../../plugins/fusion/scripts/lib/engine-acceptance.mjs"; +import { readEngineJobRecord, resolveEngineStateRoots } from "../../plugins/fusion/scripts/lib/engine-job-state.mjs"; + +const DEFAULT_TASK_ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "tasks"); +const CURSOR_PLUGIN_FILE = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..", "plugins", "cursor", ".claude-plugin", "plugin.json"); +const ENGINES = new Set(["codex", "grok", "cursor"]); +const MODES = new Set(["write", "consult"]); +const KILL_GRACE_MS = 20000; +const OPTIONS = new Set(["task", "engine", "mode", "repetition", "results", "model", "effort", "brief", "fixtures", "task-root", "timeout-ms"]); +const USAGE = "Usage: node bench/lanes/run.mjs --task --engine --mode --repetition --results [--model ] [--effort ] [--brief ] [--fixtures ] [--task-root ] [--timeout-ms ]"; + +function parseArgs(argv) { + const raw = {}; + for (let index = 0; index < argv.length; index += 2) { + const option = argv[index]; + const value = argv[index + 1]; + if (!option?.startsWith("--") || !OPTIONS.has(option.slice(2)) || value === undefined || value.startsWith("--")) { + throw new Error(`Invalid or incomplete option: ${option ?? ""}`); + } + if (Object.hasOwn(raw, option.slice(2))) { + throw new Error(`Duplicate option: ${option}`); + } + raw[option.slice(2)] = value; + } + for (const key of ["task", "engine", "mode", "repetition", "results"]) { + if (!raw[key]) { + throw new Error(`Missing required option --${key}`); + } + } + if (!ENGINES.has(raw.engine) || !MODES.has(raw.mode)) { + throw new Error("Unknown engine or mode"); + } + if (!/^[1-9]\d*$/.test(raw.repetition) || !Number.isSafeInteger(Number(raw.repetition))) { + throw new Error("Repetition must be a positive integer"); + } + if (raw["timeout-ms"] !== undefined && (!/^[1-9]\d*$/.test(raw["timeout-ms"]) || !Number.isSafeInteger(Number(raw["timeout-ms"])))) { + throw new Error("Timeout must be a positive integer"); + } + if (raw.mode === "consult" && !raw.brief) { + throw new Error("--brief is required in consult mode"); + } + for (const key of ["model", "effort", "brief", "fixtures", "task-root"]) { + if (raw[key] === "") { + throw new Error(`Empty value for --${key}`); + } + } + return { + taskId: raw.task, + engine: raw.engine, + mode: raw.mode, + repetition: Number(raw.repetition), + resultsDir: path.resolve(raw.results), + model: raw.model ?? null, + effort: raw.effort ?? null, + brief: raw.brief ? path.resolve(raw.brief) : null, + fixtures: raw.fixtures ?? "fixtures", + taskRoot: path.resolve(raw["task-root"] ?? DEFAULT_TASK_ROOT), + timeoutMs: Number(raw["timeout-ms"] ?? 900000) + }; +} + +function inside(parent, target) { + const relative = path.relative(parent, target); + return relative !== "" && relative !== ".." && !relative.startsWith(`..${path.sep}`) && !path.isAbsolute(relative); +} + +function taskFiles(options) { + const taskDir = path.resolve(options.taskRoot, options.taskId); + const fixturesDir = path.resolve(taskDir, options.fixtures); + if (!inside(options.taskRoot, taskDir) || !inside(taskDir, fixturesDir)) { + throw new Error("Task or fixtures path escapes its root"); + } + const briefFile = options.brief ?? path.join(taskDir, "brief.md"); + const verifyFile = path.join(taskDir, "verify.sh"); + if (!fs.statSync(taskDir).isDirectory() || !fs.statSync(fixturesDir).isDirectory() || !fs.statSync(briefFile).isFile() || (options.mode === "write" && !fs.statSync(verifyFile).isFile())) { + throw new Error("Task files are missing"); + } + if (!inside(fs.realpathSync(taskDir), fs.realpathSync(fixturesDir))) { + throw new Error("Fixtures path escapes task through a symlink"); + } + return { fixturesDir, briefText: fs.readFileSync(briefFile, "utf8"), verifyFile }; +} + +function git(workspace, args) { + const result = spawnSync("git", args, { cwd: workspace, encoding: "utf8" }); + if (result.error || result.status !== 0) { + throw new Error(`git ${args[0]} failed: ${result.error?.message ?? result.stderr?.trim() ?? result.status}`); + } + return result.stdout; +} + +function prepareWorkspace(fixturesDir) { + const tempRoot = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), "fusion-bench-lane-"))); + const workspace = path.join(tempRoot, "work"); + try { + fs.cpSync(fixturesDir, workspace, { recursive: true }); + git(workspace, ["init", "-q"]); + git(workspace, ["config", "user.name", "Fusion Bench"]); + git(workspace, ["config", "user.email", "fusion-bench@localhost"]); + git(workspace, ["add", "-A"]); + git(workspace, ["commit", "-q", "--allow-empty", "-m", "benchmark fixture"]); + return { tempRoot, workspace }; + } catch (error) { + fs.rmSync(tempRoot, { recursive: true, force: true }); + throw error; + } +} + +function runCompanion(companion, args, workspace, timeoutMs) { + return new Promise((resolve) => { + const started = process.hrtime.bigint(); + let child; + try { + child = spawn(process.execPath, [companion, ...args], { cwd: workspace, env: process.env, detached: true, stdio: ["ignore", "pipe", "pipe"] }); + } catch (error) { + resolve({ stdout: "", exit: null, seconds: 0, failure: { type: "start", message: error.message } }); + return; + } + let stdout = ""; + let stderr = ""; + let timedOut = false; + let startError = null; + child.stdout.on("data", (chunk) => { stdout += chunk; }); + child.stderr.on("data", (chunk) => { stderr += chunk; }); + child.on("error", (error) => { startError = error; }); + const signalGroup = (signal) => { + try { + process.kill(-child.pid, signal); + } catch { + child.kill(signal); + } + }; + let killTimer = null; + const timer = setTimeout(() => { + timedOut = true; + signalGroup("SIGTERM"); + killTimer = setTimeout(() => signalGroup("SIGKILL"), KILL_GRACE_MS); + }, timeoutMs); + child.on("close", (code) => { + clearTimeout(timer); + clearTimeout(killTimer); + resolve({ + stdout, + exit: code, + seconds: Number(process.hrtime.bigint() - started) / 1e9, + failure: timedOut ? { type: "timeout", message: `Companion timed out after ${timeoutMs} ms` } : startError ? { type: "start", message: startError.message } : null, + stderr + }); + }); + }); +} + +function jobIdFromStdout(stdout, engine) { + try { + const payload = JSON.parse(stdout); + const id = engine === "codex" ? payload?.id : payload?.jobId; + return typeof id === "string" && /^[0-9a-f]{32}$/.test(id) ? id : null; + } catch { + return null; + } +} + +function failedJobIdFromState(engine, workspace) { + for (const root of resolveEngineStateRoots(engine, process.env)) { + let workspaces; + try { + workspaces = fs.readdirSync(root, { withFileTypes: true }); + } catch { + continue; + } + for (const entry of workspaces) { + if (!entry.isDirectory()) { + continue; + } + let jobs; + try { + jobs = fs.readdirSync(path.join(root, entry.name, "jobs")); + } catch { + continue; + } + for (const filename of jobs) { + const id = filename.endsWith(".json") ? filename.slice(0, -5) : ""; + if (!/^[0-9a-f]{32}$/.test(id)) { + continue; + } + try { + const record = JSON.parse(fs.readFileSync(path.join(root, entry.name, "jobs", filename), "utf8")); + if (record?.cwd === workspace) { + return id; + } + } catch { + continue; + } + } + } + } + return null; +} + +function writeEnvOnce(resultsDir, taskManifestHash) { + const envFile = path.join(resultsDir, "env.json"); + if (fs.existsSync(envFile)) { + return; + } + const env = { + date: new Date().toISOString(), + nodeVersion: process.version, + os: { platform: os.platform(), release: os.release() }, + pluginVersions: { ...collectPluginVersions(process.env), cursor: readJsonFile(CURSOR_PLUGIN_FILE)?.version ?? null }, + engineCliVersions: { codex: cliVersion("codex"), grok: cliVersion("grok"), cursor: cliVersion("cursor-agent") }, + taskManifestHash, + manifestHash: taskManifestHash + }; + try { + fs.writeFileSync(envFile, `${JSON.stringify(env, null, 2)}\n`, { flag: "wx" }); + } catch (error) { + if (error.code !== "EEXIST") { + throw error; + } + } +} + +async function attempt(options) { + const { fixturesDir, briefText, verifyFile } = taskFiles(options); + const taskManifestHash = buildManifest(options.taskRoot).manifestHash; + const runId = randomUUID(); + const startedAt = new Date().toISOString(); + const { tempRoot, workspace } = prepareWorkspace(fixturesDir); + try { + fs.mkdirSync(options.resultsDir, { recursive: true }); + writeEnvOnce(options.resultsDir, taskManifestHash); + const companionPath = newestEngineCompanion(options.engine, process.env); + const args = ["task", ...(options.mode === "write" ? ["--write"] : []), "--json", ...(options.model ? ["--model", options.model] : []), ...(options.effort ? ["--effort", options.effort] : []), "--cwd", workspace, "--", briefText]; + const outcome = companionPath ? await runCompanion(companionPath, args, workspace, options.timeoutMs) : { stdout: "", exit: null, seconds: 0, failure: { type: "start", message: `No ${options.engine} companion found` } }; + const jobId = outcome.failure ? null : jobIdFromStdout(outcome.stdout, options.engine) ?? (outcome.exit !== 0 ? failedJobIdFromState(options.engine, workspace) : null); + const job = jobId ? readEngineJobRecord(options.engine, jobId, process.env) : null; + const infrastructureFailure = outcome.failure ?? (!jobId ? { type: "job_id", message: "Companion printed no parseable job id" } : !job ? { type: "job_record", message: `Job record ${jobId} could not be read` } : null); + let verifyExit = null; + let verdict = infrastructureFailure ? "infra_failure" : options.mode === "consult" ? "unscored" : "fail"; + let resultFile = null; + let diffFile = null; + if (!infrastructureFailure && options.mode === "write") { + git(workspace, ["add", "-A"]); + diffFile = path.join(options.resultsDir, `${runId}.diff`); + fs.writeFileSync(diffFile, git(workspace, ["diff", "--cached"])); + const verify = spawnSync("bash", [verifyFile, workspace], { cwd: options.resultsDir, encoding: "utf8" }); + verifyExit = verify.status ?? 1; + verdict = verifyExit === 0 && outcome.exit === 0 && job.status === "done" && !job.failureKind ? "pass" : "fail"; + } else if (!infrastructureFailure) { + resultFile = path.join(options.resultsDir, `${runId}.result.md`); + fs.writeFileSync(resultFile, job.resultText ?? "", "utf8"); + } + const record = { + schema: "fusion-bench-lane/v1", + runId, + taskId: options.taskId, + mode: options.mode, + engine: options.engine, + model: options.model, + effort: options.effort, + repetition: options.repetition, + startedAt, + finishedAt: new Date().toISOString(), + wallClockSeconds: outcome.seconds, + companionPath, + companionExit: outcome.exit, + jobId, + jobState: job?.status ?? null, + failureKind: job?.failureKind ?? null, + resolvedModel: job?.resolvedModel ?? job?.observedModel ?? null, + resolvedEffort: job?.resolvedEffort ?? null, + billingPool: job?.billingPool ?? null, + cliVersion: job?.[`${options.engine}Version`] ?? null, + usage: job?.usage ?? job?.cumulativeTokenUsage ?? null, + totalCostUsd: job?.totalCostUsd ?? null, + verifyExit, + verdict, + infrastructureFailure, + resultFile, + diffFile, + taskManifestHash + }; + fs.appendFileSync(path.join(options.resultsDir, "runs.jsonl"), `${JSON.stringify(record)}\n`); + return record; + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } +} + +let options; +try { + options = parseArgs(process.argv.slice(2)); +} catch (error) { + process.stderr.write(`${error.message}\n${USAGE}\n`); + process.exitCode = 2; +} +if (options) { + attempt(options).catch((error) => { + process.stderr.write(`${error.message}\n`); + process.exitCode = 1; + }); +} diff --git a/bench/lanes/summarize.mjs b/bench/lanes/summarize.mjs new file mode 100644 index 0000000..34cd14c --- /dev/null +++ b/bench/lanes/summarize.mjs @@ -0,0 +1,90 @@ +#!/usr/bin/env node + +import fs from "node:fs"; +import path from "node:path"; + +function parseArgs(argv) { + let results = null; + let json = false; + for (let index = 0; index < argv.length; index += 1) { + if (argv[index] === "--json" && !json) { + json = true; + } else if (argv[index] === "--results" && !results && argv[index + 1] && !argv[index + 1].startsWith("--")) { + results = path.resolve(argv[++index]); + } else { + throw new Error(`Invalid option: ${argv[index]}`); + } + } + if (!results) { + throw new Error("Missing --results"); + } + return { results, json }; +} + +function median(values) { + const sorted = values.filter((value) => typeof value === "number" && Number.isFinite(value)).sort((a, b) => a - b); + if (!sorted.length) { + return null; + } + const middle = Math.floor(sorted.length / 2); + return sorted.length % 2 ? sorted[middle] : (sorted[middle - 1] + sorted[middle]) / 2; +} + +function tokens(usage, camel, snake) { + return usage?.[camel] ?? usage?.[snake] ?? null; +} + +function summarize(records) { + const groups = new Map(); + for (const record of records) { + if (record.schema !== "fusion-bench-lane/v1") { + throw new Error(`Unexpected record schema: ${record.schema}`); + } + const key = JSON.stringify([record.taskId, record.engine, record.model, record.effort]); + if (!groups.has(key)) { + groups.set(key, []); + } + groups.get(key).push(record); + } + return [...groups].sort(([left], [right]) => left.localeCompare(right)).map(([key, runs]) => { + const [taskId, engine, model, effort] = JSON.parse(key); + const kinds = new Map(); + for (const run of runs) { + if (run.failureKind) { + kinds.set(run.failureKind, (kinds.get(run.failureKind) ?? 0) + 1); + } + } + return { + taskId, + engine, + model, + effort, + runs: runs.length, + pass: runs.filter((run) => run.verdict === "pass").length, + fail: runs.filter((run) => run.verdict === "fail").length, + infra: runs.filter((run) => run.verdict === "infra_failure").length, + unscored: runs.filter((run) => run.verdict === "unscored").length, + medianWallClockSeconds: median(runs.map((run) => run.wallClockSeconds)), + medianInputTokens: median(runs.map((run) => tokens(run.usage, "inputTokens", "input_tokens"))), + medianOutputTokens: median(runs.map((run) => tokens(run.usage, "outputTokens", "output_tokens"))), + failureKinds: [...kinds].sort(([a], [b]) => a.localeCompare(b)).map(([kind, count]) => `${kind}:${count}`).join(", ") + }; + }); +} + +function markdown(rows) { + const columns = ["taskId", "engine", "model", "effort", "runs", "pass", "fail", "infra", "unscored", "medianWallClockSeconds", "medianInputTokens", "medianOutputTokens", "failureKinds"]; + const headers = ["taskId", "engine", "model", "effort", "runs", "pass", "fail", "infra", "unscored", "median wall clock seconds", "median input tokens", "median output tokens", "failure kinds"]; + const cell = (value) => String(value ?? "").replaceAll("|", "\\|").replaceAll("\n", " "); + return [`| ${headers.join(" | ")} |`, `| ${headers.map(() => "---").join(" | ")} |`, ...rows.map((row) => `| ${columns.map((column) => cell(row[column])).join(" | ")} |`)].join("\n"); +} + +try { + const options = parseArgs(process.argv.slice(2)); + const records = fs.readFileSync(path.join(options.results, "runs.jsonl"), "utf8").trim().split("\n").filter(Boolean).map((line) => JSON.parse(line)); + const rows = summarize(records); + process.stdout.write(`${options.json ? JSON.stringify(rows, null, 2) : markdown(rows)}\n`); +} catch (error) { + process.stderr.write(`${error.message}\n`); + process.exitCode = 1; +} diff --git a/bench/lib.mjs b/bench/lib.mjs new file mode 100644 index 0000000..31c1446 --- /dev/null +++ b/bench/lib.mjs @@ -0,0 +1,79 @@ +import { spawnSync } from "node:child_process"; +import fs from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +const REPO_ROOT = path.dirname(path.dirname(fileURLToPath(import.meta.url))); +const CLI_VERSION_TIMEOUT_MS = 5000; +const PLUGIN_VERSION_FILES = [ + { name: "codex", file: path.join(REPO_ROOT, "plugins", "codex", ".claude-plugin", "plugin.json") }, + { name: "fusion", file: path.join(REPO_ROOT, "plugins", "fusion", ".claude-plugin", "plugin.json") }, + { name: "grok", file: path.join(REPO_ROOT, "plugins", "grok", ".claude-plugin", "plugin.json") } +]; + +export function readJsonFile(file) { + if (!file || !fs.existsSync(file)) { + return null; + } + try { + return JSON.parse(fs.readFileSync(file, "utf8")); + } catch { + return null; + } +} + +export function collectPluginVersions(env = process.env) { + const versions = {}; + const marketplace = readJsonFile(path.join(REPO_ROOT, ".claude-plugin", "marketplace.json")); + const marketplaceVersion = marketplace?.version ?? marketplace?.metadata?.version; + versions["claude-code-fusion"] = typeof marketplaceVersion === "string" ? marketplaceVersion : null; + for (const source of PLUGIN_VERSION_FILES) { + const plugin = readJsonFile(source.file); + versions[source.name] = typeof plugin?.version === "string" ? plugin.version : null; + } + return versions; +} + +function hasPathSeparator(bin) { + return bin.includes("/") || (path.sep === "\\" && bin.includes("\\")); +} + +function commandOnPath(bin) { + if (hasPathSeparator(bin)) { + try { + fs.accessSync(bin, fs.constants.X_OK); + return true; + } catch { + return false; + } + } + const pathValue = process.env.PATH ?? ""; + return pathValue.split(path.delimiter).some((entry) => { + if (!entry) { + return false; + } + try { + fs.accessSync(path.join(entry, bin), fs.constants.X_OK); + return true; + } catch { + return false; + } + }); +} + +export function cliVersion(bin) { + if (!commandOnPath(bin)) { + return null; + } + const result = spawnSync(bin, ["--version"], { + cwd: REPO_ROOT, + env: process.env, + encoding: "utf8", + timeout: CLI_VERSION_TIMEOUT_MS + }); + if (result.error || result.status !== 0) { + return null; + } + const text = `${result.stdout ?? ""}${result.stderr ?? ""}`.trim(); + return text ? text.split(/\r?\n/)[0].trim() : null; +} diff --git a/bench/run.mjs b/bench/run.mjs index 098fb06..26ed6b6 100644 --- a/bench/run.mjs +++ b/bench/run.mjs @@ -1,17 +1,17 @@ #!/usr/bin/env node import { randomUUID } from "node:crypto"; -import { spawn, spawnSync } from "node:child_process"; +import { spawn } from "node:child_process"; import fs from "node:fs"; import os from "node:os"; import path from "node:path"; import { performance } from "node:perf_hooks"; import { fileURLToPath } from "node:url"; import { buildManifest } from "./manifest.mjs"; +import { cliVersion, collectPluginVersions, readJsonFile } from "./lib.mjs"; const SELF_PATH = fileURLToPath(import.meta.url); const BENCH_DIR = path.dirname(SELF_PATH); -const REPO_ROOT = path.dirname(BENCH_DIR); const DEFAULT_TASK_ROOT = path.join(BENCH_DIR, "tasks"); const CONDITIONS = new Set(["A", "B1", "B2", "B3"]); const CONDITION_PLUGIN_IDS = ["codex@claude-code-fusion", "codex@openai-codex", "fusion@claude-code-fusion", "grok@claude-code-fusion"]; @@ -21,12 +21,6 @@ const EXPECTED_CONDITION_PLUGINS = { B2: ["codex@claude-code-fusion", "fusion@claude-code-fusion", "grok@claude-code-fusion"], B3: ["fusion@claude-code-fusion"] }; -const CLI_VERSION_TIMEOUT_MS = 5000; -const PLUGIN_VERSION_FILES = [ - { name: "codex", file: path.join(REPO_ROOT, "plugins", "codex", ".claude-plugin", "plugin.json") }, - { name: "fusion", file: path.join(REPO_ROOT, "plugins", "fusion", ".claude-plugin", "plugin.json") }, - { name: "grok", file: path.join(REPO_ROOT, "plugins", "grok", ".claude-plugin", "plugin.json") } -]; const TOKEN_FIELDS = [ "input_tokens", "output_tokens", @@ -189,76 +183,6 @@ function runProcess(bin, args, options) { }); } -function readJsonFile(file) { - if (!file) { - return null; - } - if (!fs.existsSync(file)) { - return null; - } - try { - return JSON.parse(fs.readFileSync(file, "utf8")); - } catch { - return null; - } -} - -function collectPluginVersions(env = process.env) { - const versions = {}; - const marketplace = readJsonFile(path.join(REPO_ROOT, ".claude-plugin", "marketplace.json")); - const marketplaceVersion = marketplace?.version ?? marketplace?.metadata?.version; - versions["claude-code-fusion"] = typeof marketplaceVersion === "string" ? marketplaceVersion : null; - for (const source of PLUGIN_VERSION_FILES) { - const plugin = readJsonFile(source.file); - versions[source.name] = typeof plugin?.version === "string" ? plugin.version : null; - } - return versions; -} - -function hasPathSeparator(bin) { - return bin.includes("/") || (path.sep === "\\" && bin.includes("\\")); -} - -function commandOnPath(bin) { - if (hasPathSeparator(bin)) { - try { - fs.accessSync(bin, fs.constants.X_OK); - return true; - } catch { - return false; - } - } - const pathValue = process.env.PATH ?? ""; - return pathValue.split(path.delimiter).some((entry) => { - if (!entry) { - return false; - } - try { - fs.accessSync(path.join(entry, bin), fs.constants.X_OK); - return true; - } catch { - return false; - } - }); -} - -function cliVersion(bin) { - if (!commandOnPath(bin)) { - return null; - } - const result = spawnSync(bin, ["--version"], { - cwd: REPO_ROOT, - env: process.env, - encoding: "utf8", - timeout: CLI_VERSION_TIMEOUT_MS - }); - if (result.error || result.status !== 0) { - return null; - } - const text = `${result.stdout ?? ""}${result.stderr ?? ""}`.trim(); - return text ? text.split(/\r?\n/)[0].trim() : null; -} - function collectEngineCliVersions(options) { return { claude: cliVersion(options.claudeBin), diff --git a/tests/bench-lanes.test.mjs b/tests/bench-lanes.test.mjs new file mode 100644 index 0000000..7965f9a --- /dev/null +++ b/tests/bench-lanes.test.mjs @@ -0,0 +1,203 @@ +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { test } from "node:test"; + +const repoRoot = path.join(import.meta.dirname, ".."); +const runner = path.join(repoRoot, "bench", "lanes", "run.mjs"); +const summarizer = path.join(repoRoot, "bench", "lanes", "summarize.mjs"); +const fixtureRoot = path.join(import.meta.dirname, "fixtures", "bench-lanes"); +const companion = path.join(fixtureRoot, "fake-companion.mjs"); + +function sandbox(t) { + const root = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), "bench-lanes-test-"))); + t.after(() => fs.rmSync(root, { recursive: true, force: true })); + const bins = path.join(root, "bins"); + fs.mkdirSync(bins); + for (const name of ["codex", "grok", "cursor"]) { + const file = path.join(bins, name); + fs.writeFileSync(file, "#!/usr/bin/env bash\nprintf '%s\\n' 'fake version'\n"); + fs.chmodSync(file, 0o755); + } + return { + root, + taskRoot: path.join(fixtureRoot, "tasks"), + results: path.join(root, "results"), + env: { + ...process.env, + PATH: `${bins}${path.delimiter}${process.env.PATH}`, + FUSION_CODEX_COMPANION: companion, + FUSION_GROK_COMPANION: companion, + FUSION_CURSOR_COMPANION: companion, + CODEX_COMPANION_DATA: path.join(root, "codex-data"), + GROK_COMPANION_DATA: path.join(root, "grok-data"), + CURSOR_COMPANION_DATA: path.join(root, "cursor-data"), + FAKE_VERIFY_LOG: path.join(root, "verify.log"), + FAKE_VERIFY_FAIL: "0", + FAKE_ARGS_FILE: path.join(root, "args.json"), + FAKE_PID_FILE: path.join(root, "pid") + } + }; +} + +function run(s, { engine = "codex", mode = "write", fakeMode = "pass", extra = [], env = {} } = {}) { + const args = [runner, "--task", "T01-test", "--engine", engine, "--mode", mode, "--repetition", "1", "--results", s.results, "--task-root", s.taskRoot, "--model", "requested-model", "--effort", "xhigh", ...(mode === "consult" ? ["--brief", path.join(s.taskRoot, "T01-test", "brief.md")] : []), ...extra]; + return spawnSync(process.execPath, args, { cwd: repoRoot, env: { ...s.env, FAKE_ENGINE: engine, FAKE_MODE: fakeMode, ...env }, encoding: "utf8", timeout: 30000 }); +} + +function records(s) { + return fs.readFileSync(path.join(s.results, "runs.jsonl"), "utf8").trim().split("\n").map((line) => JSON.parse(line)); +} + +test("write pass records a staged diff and requested lane", (t) => { + const s = sandbox(t); + const result = run(s); + assert.equal(result.status, 0, result.stderr); + const [record] = records(s); + assert.deepEqual(Object.keys(record), ["schema", "runId", "taskId", "mode", "engine", "model", "effort", "repetition", "startedAt", "finishedAt", "wallClockSeconds", "companionPath", "companionExit", "jobId", "jobState", "failureKind", "resolvedModel", "resolvedEffort", "billingPool", "cliVersion", "usage", "totalCostUsd", "verifyExit", "verdict", "infrastructureFailure", "resultFile", "diffFile", "taskManifestHash"]); + assert.equal(record.verdict, "pass"); + assert.equal(record.verifyExit, 0); + assert.equal(record.model, "requested-model"); + assert.equal(record.resolvedModel, "observed-model"); + assert.equal(record.cliVersion, "1.2.3"); + assert.match(fs.readFileSync(record.diffFile, "utf8"), /\+fake companion completed/); + const invocation = JSON.parse(fs.readFileSync(s.env.FAKE_ARGS_FILE, "utf8")); + assert.equal(invocation.cwd, invocation.args[invocation.args.indexOf("--cwd") + 1]); + assert.deepEqual(invocation.args.slice(0, 8), ["task", "--write", "--json", "--model", "requested-model", "--effort", "xhigh", "--cwd"]); + assert.equal(invocation.args.at(-1), "Edit the marker file.\n"); + assert.equal(fs.existsSync(invocation.cwd), false); +}); + +test("write verifier failure is a scored fail", (t) => { + const s = sandbox(t); + const result = run(s, { env: { FAKE_VERIFY_FAIL: "1" } }); + assert.equal(result.status, 0, result.stderr); + assert.equal(records(s)[0].verdict, "fail"); + assert.equal(records(s)[0].verifyExit, 10); +}); + +test("engine failure with a job record is scored, not infrastructure failure", (t) => { + const s = sandbox(t); + const result = run(s, { fakeMode: "quota" }); + assert.equal(result.status, 0, result.stderr); + const [record] = records(s); + assert.equal(record.verdict, "fail"); + assert.equal(record.failureKind, "quota"); + assert.equal(record.infrastructureFailure, null); + assert.equal(record.companionExit, 1); + assert.equal(record.verifyExit, 9); +}); + +test("companion error output still uses its workspace job record", (t) => { + const s = sandbox(t); + const result = run(s, { engine: "grok", fakeMode: "stderr-error" }); + assert.equal(result.status, 0, result.stderr); + const [record] = records(s); + assert.equal(record.verdict, "fail"); + assert.equal(record.failureKind, "quota"); + assert.equal(record.infrastructureFailure, null); + assert.equal(record.jobId, "a".repeat(32)); +}); + +test("consult saves the result and skips verification", (t) => { + const s = sandbox(t); + const result = run(s, { engine: "grok", mode: "consult" }); + assert.equal(result.status, 0, result.stderr); + const [record] = records(s); + assert.equal(record.verdict, "unscored"); + assert.equal(record.verifyExit, null); + assert.equal(record.diffFile, null); + assert.equal(fs.readFileSync(record.resultFile, "utf8"), "consult result\n"); + assert.equal(fs.existsSync(s.env.FAKE_VERIFY_LOG), false); +}); + +test("Cursor consult reads its job id and preserves engine failure", (t) => { + const s = sandbox(t); + const result = run(s, { engine: "cursor", mode: "consult", fakeMode: "quota" }); + assert.equal(result.status, 0, result.stderr); + assert.equal(records(s)[0].verdict, "unscored"); + assert.equal(records(s)[0].failureKind, "quota"); +}); + +test("unparseable stdout appends an infrastructure failure", (t) => { + const s = sandbox(t); + const result = run(s, { fakeMode: "nojson" }); + assert.equal(result.status, 0, result.stderr); + const [record] = records(s); + assert.equal(record.verdict, "infra_failure"); + assert.equal(record.infrastructureFailure.type, "job_id"); + assert.equal(record.jobId, null); +}); + +test("missing job record appends an infrastructure failure", (t) => { + const s = sandbox(t); + const result = run(s, { fakeMode: "missing-record" }); + assert.equal(result.status, 0, result.stderr); + assert.equal(records(s)[0].infrastructureFailure.type, "job_record"); +}); + +test("timeout kills the companion process group and records infra failure", (t) => { + const s = sandbox(t); + const result = run(s, { fakeMode: "timeout", extra: ["--timeout-ms", "300"] }); + assert.equal(result.status, 0, result.stderr); + const [record] = records(s); + assert.equal(record.infrastructureFailure.type, "timeout"); + assert.equal(record.verdict, "infra_failure"); + assert.equal(record.companionExit, null); + const pid = Number(fs.readFileSync(s.env.FAKE_PID_FILE, "utf8")); + assert.throws(() => process.kill(pid, 0), { code: "ESRCH" }); +}); + +test("usage errors exit 2 without writing a record", (t) => { + const s = sandbox(t); + for (const extra of [["--bogus", "x"], ["--model"], ["--repetition", "0"], ["--engine", "other"], ["--mode", "other"]]) { + const args = [runner, "--task", "T01-test", "--engine", "codex", "--mode", "write", "--repetition", "1", "--results", s.results, ...extra]; + const result = spawnSync(process.execPath, args, { cwd: repoRoot, env: s.env, encoding: "utf8" }); + assert.equal(result.status, 2, result.stderr); + assert.match(result.stderr, /Usage:/); + } + const consult = run(s, { mode: "consult", extra: ["--brief"] }); + assert.equal(consult.status, 2); + assert.equal(fs.existsSync(path.join(s.results, "runs.jsonl")), false); +}); + +test("env.json is written once", (t) => { + const s = sandbox(t); + assert.equal(run(s).status, 0); + const file = path.join(s.results, "env.json"); + const initial = fs.readFileSync(file, "utf8"); + const env = JSON.parse(initial); + assert.equal(env.engineCliVersions.codex, "fake version"); + assert.equal(env.pluginVersions.cursor, JSON.parse(fs.readFileSync(path.join(repoRoot, "plugins", "cursor", ".claude-plugin", "plugin.json"), "utf8")).version); + assert.equal(env.os.platform, process.platform); + assert.equal(env.taskManifestHash, records(s)[0].taskManifestHash); + assert.equal(run(s, { engine: "grok", mode: "consult" }).status, 0); + assert.equal(fs.readFileSync(file, "utf8"), initial); + assert.equal(records(s).length, 2); +}); + +test("summarizer groups records and normalizes token shapes", (t) => { + const s = sandbox(t); + assert.equal(run(s).status, 0); + assert.equal(run(s, { fakeMode: "quota" }).status, 0); + assert.equal(run(s, { engine: "grok", mode: "consult" }).status, 0); + const json = spawnSync(process.execPath, [summarizer, "--results", s.results, "--json"], { cwd: repoRoot, encoding: "utf8" }); + assert.equal(json.status, 0, json.stderr); + const rows = JSON.parse(json.stdout); + assert.equal(rows.length, 2); + const codex = rows.find((row) => row.engine === "codex"); + assert.equal(codex.runs, 2); + assert.equal(codex.pass, 1); + assert.equal(codex.fail, 1); + assert.equal(codex.medianInputTokens, 10); + assert.equal(codex.failureKinds, "quota:1"); + const grok = rows.find((row) => row.engine === "grok"); + assert.equal(grok.medianInputTokens, 12); + assert.equal(grok.medianOutputTokens, 8); + const table = spawnSync(process.execPath, [summarizer, "--results", s.results], { cwd: repoRoot, encoding: "utf8" }); + assert.equal(table.status, 0, table.stderr); + assert.match(table.stdout, /median wall clock seconds/); + assert.match(table.stdout, /quota:1/); +}); diff --git a/tests/fixtures/bench-lanes/fake-companion.mjs b/tests/fixtures/bench-lanes/fake-companion.mjs new file mode 100644 index 0000000..4ee38b6 --- /dev/null +++ b/tests/fixtures/bench-lanes/fake-companion.mjs @@ -0,0 +1,48 @@ +import { spawn } from "node:child_process"; +import fs from "node:fs"; +import path from "node:path"; + +const args = process.argv.slice(2); +const value = (name) => args[args.indexOf(name) + 1]; +const engine = process.env.FAKE_ENGINE; +const mode = process.env.FAKE_MODE ?? "pass"; +const workspace = value("--cwd"); +const jobId = "a".repeat(32); + +fs.writeFileSync(process.env.FAKE_ARGS_FILE, JSON.stringify({ args, cwd: process.cwd() })); +if (mode === "timeout") { + fs.writeFileSync(process.env.FAKE_PID_FILE, String(process.pid)); + spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)"], { stdio: "ignore" }); + setInterval(() => {}, 1000); +} else if (mode === "nojson") { + process.stdout.write("no job id\n"); +} else if (mode === "missing-record") { + process.stdout.write(`${JSON.stringify(engine === "codex" ? { id: jobId } : { jobId })}\n`); +} else { + if (args.includes("--write") && mode !== "quota" && mode !== "stderr-error") { + fs.writeFileSync(path.join(workspace, "marker.txt"), "fake companion completed\n"); + } + const record = { + cwd: workspace, + status: mode === "quota" || mode === "stderr-error" ? "error" : "done", + failureKind: mode === "quota" || mode === "stderr-error" ? "quota" : null, + resultText: "consult result\n", + observedModel: "observed-model", + resolvedEffort: "high", + billingPool: "included", + [`${engine}Version`]: "1.2.3", + usage: engine === "grok" ? { input_tokens: 12, output_tokens: 8, cache_read_input_tokens: 99 } : { inputTokens: 10, outputTokens: 6, cachedInputTokens: 99 }, + totalCostUsd: 0 + }; + const jobFile = path.join(process.env[`${engine.toUpperCase()}_COMPANION_DATA`], "state", "workspace", "jobs", `${jobId}.json`); + fs.mkdirSync(path.dirname(jobFile), { recursive: true }); + fs.writeFileSync(jobFile, JSON.stringify(record)); + if (mode === "stderr-error") { + process.stderr.write(`${JSON.stringify({ status: "error", failureKind: "quota", message: `job: ${jobId}` })}\n`); + } else { + process.stdout.write(`${JSON.stringify(engine === "codex" ? { id: jobId } : { jobId })}\n`); + } + if (mode === "quota" || mode === "stderr-error") { + process.exitCode = 1; + } +} diff --git a/tests/fixtures/bench-lanes/tasks/T01-test/brief.md b/tests/fixtures/bench-lanes/tasks/T01-test/brief.md new file mode 100644 index 0000000..5c77a4a --- /dev/null +++ b/tests/fixtures/bench-lanes/tasks/T01-test/brief.md @@ -0,0 +1 @@ +Edit the marker file. diff --git a/tests/fixtures/bench-lanes/tasks/T01-test/fixtures/input.txt b/tests/fixtures/bench-lanes/tasks/T01-test/fixtures/input.txt new file mode 100644 index 0000000..ee8c1ee --- /dev/null +++ b/tests/fixtures/bench-lanes/tasks/T01-test/fixtures/input.txt @@ -0,0 +1 @@ +fixture diff --git a/tests/fixtures/bench-lanes/tasks/T01-test/verify.sh b/tests/fixtures/bench-lanes/tasks/T01-test/verify.sh new file mode 100644 index 0000000..c4ac73c --- /dev/null +++ b/tests/fixtures/bench-lanes/tasks/T01-test/verify.sh @@ -0,0 +1,6 @@ +#!/usr/bin/env bash +printf '%s\n' "$1" >> "$FAKE_VERIFY_LOG" +test "$PWD" != "$1" || exit 7 +test ! -e "$1/verify.sh" || exit 8 +test -f "$1/marker.txt" || exit 9 +test "$FAKE_VERIFY_FAIL" != 1 || exit 10 From 3d19401998f7dd43471f65600da33fef35e0028a Mon Sep 17 00:00:00 2001 From: Harry Yep Date: Tue, 13 Oct 2026 17:25:49 +0800 Subject: [PATCH 4/8] docs(bench): record the 2026-10-13 lane snapshot --- bench/results/lanes-2026-10-13/env.json | 22 ++++++++++++ bench/results/lanes-2026-10-13/runs.jsonl | 44 +++++++++++++++++++++++ bench/results/lanes-2026-10-13/summary.md | 43 ++++++++++++++++++++++ 3 files changed, 109 insertions(+) create mode 100644 bench/results/lanes-2026-10-13/env.json create mode 100644 bench/results/lanes-2026-10-13/runs.jsonl create mode 100644 bench/results/lanes-2026-10-13/summary.md diff --git a/bench/results/lanes-2026-10-13/env.json b/bench/results/lanes-2026-10-13/env.json new file mode 100644 index 0000000..dd9e04b --- /dev/null +++ b/bench/results/lanes-2026-10-13/env.json @@ -0,0 +1,22 @@ +{ + "date": "2026-10-13T08:43:24.471Z", + "nodeVersion": "v24.21.0", + "os": { + "platform": "darwin", + "release": "27.0.0" + }, + "pluginVersions": { + "claude-code-fusion": "0.0.63", + "codex": "0.0.63", + "fusion": "0.0.63", + "grok": "0.0.63", + "cursor": "0.0.63" + }, + "engineCliVersions": { + "codex": "codex-cli 0.162.1", + "grok": "grok 1.0.52 (0920c8169e27) [stable]", + "cursor": "2026.10.01-e373342" + }, + "taskManifestHash": "e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967", + "manifestHash": "e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967" +} diff --git a/bench/results/lanes-2026-10-13/runs.jsonl b/bench/results/lanes-2026-10-13/runs.jsonl new file mode 100644 index 0000000..c77a6be --- /dev/null +++ b/bench/results/lanes-2026-10-13/runs.jsonl @@ -0,0 +1,44 @@ +{"schema":"fusion-bench-lane/v1","runId":"c8ad0e45-148a-4ab7-8498-70df284caf61","taskId":"T05-mechanical-codemod","mode":"write","engine":"cursor","model":"composer-2.5","effort":"default","repetition":1,"startedAt":"2026-10-13T08:43:24.390Z","finishedAt":"2026-10-13T08:44:36.059Z","wallClockSeconds":71.2012325,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"46b9fec736e6061956cc796a1ecc8906","jobState":"done","failureKind":null,"resolvedModel":"composer-2.5","resolvedEffort":null,"billingPool":"first-party","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":11365,"outputTokens":3022,"cacheReadTokens":45636,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":0,"verdict":"pass","infrastructureFailure":null,"resultFile":null,"diffFile":"c8ad0e45-148a-4ab7-8498-70df284caf61.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"dd2871e3-db1e-422e-a579-efd4324ec12a","taskId":"T09-style-fidelity","mode":"write","engine":"cursor","model":"composer-2.5","effort":"default","repetition":3,"startedAt":"2026-10-13T08:44:47.243Z","finishedAt":"2026-10-13T08:45:47.967Z","wallClockSeconds":60.463508958,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"4c83b3bce6dfbffc66286f97e875e47d","jobState":"done","failureKind":null,"resolvedModel":"composer-2.5","resolvedEffort":null,"billingPool":"first-party","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":38176,"outputTokens":3008,"cacheReadTokens":14816,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":0,"verdict":"pass","infrastructureFailure":null,"resultFile":null,"diffFile":"dd2871e3-db1e-422e-a579-efd4324ec12a.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"e61e62d5-35d1-45fd-a300-40e37be9bd07","taskId":"T06-plan-shaped-fixes","mode":"write","engine":"cursor","model":"composer-2.5","effort":"default","repetition":1,"startedAt":"2026-10-13T08:44:47.242Z","finishedAt":"2026-10-13T08:45:48.692Z","wallClockSeconds":61.208693917,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"c7e619a321fe2c975a699d09ae800832","jobState":"done","failureKind":null,"resolvedModel":"composer-2.5","resolvedEffort":null,"billingPool":"first-party","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":21387,"outputTokens":1521,"cacheReadTokens":19520,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":1,"verdict":"fail","infrastructureFailure":null,"resultFile":null,"diffFile":"e61e62d5-35d1-45fd-a300-40e37be9bd07.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"673a1e30-f901-475a-ab77-cda95bc41300","taskId":"T05-mechanical-codemod","mode":"write","engine":"cursor","model":"composer-2.5-fast","effort":"default","repetition":1,"startedAt":"2026-10-13T08:44:47.243Z","finishedAt":"2026-10-13T08:45:48.836Z","wallClockSeconds":61.332950042,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"90b9f4c9f6f5794dc6ccd60a30840a46","jobState":"done","failureKind":null,"resolvedModel":"composer-2.5-fast","resolvedEffort":null,"billingPool":"first-party","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":34886,"outputTokens":2693,"cacheReadTokens":21536,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":0,"verdict":"pass","infrastructureFailure":null,"resultFile":null,"diffFile":"673a1e30-f901-475a-ab77-cda95bc41300.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"06049f76-ec9d-4bc0-83d6-554570136b97","taskId":"T06-plan-shaped-fixes","mode":"write","engine":"cursor","model":"composer-2.5","effort":"default","repetition":2,"startedAt":"2026-10-13T08:44:47.242Z","finishedAt":"2026-10-13T08:45:48.911Z","wallClockSeconds":61.43524225,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"04020e404d18bed3bd5515a40c9264b4","jobState":"done","failureKind":null,"resolvedModel":"composer-2.5","resolvedEffort":null,"billingPool":"first-party","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":6247,"outputTokens":1470,"cacheReadTokens":34648,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":1,"verdict":"fail","infrastructureFailure":null,"resultFile":null,"diffFile":"06049f76-ec9d-4bc0-83d6-554570136b97.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"522a892c-08a3-4633-b8ca-5698acbb3805","taskId":"T06-plan-shaped-fixes","mode":"write","engine":"cursor","model":"composer-2.5","effort":"default","repetition":3,"startedAt":"2026-10-13T08:44:47.243Z","finishedAt":"2026-10-13T08:45:50.611Z","wallClockSeconds":63.138498042,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"4a9d65967b7c1fa614b7d5fd6e4b6774","jobState":"done","failureKind":null,"resolvedModel":"composer-2.5","resolvedEffort":null,"billingPool":"first-party","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":22517,"outputTokens":1613,"cacheReadTokens":18528,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":1,"verdict":"fail","infrastructureFailure":null,"resultFile":null,"diffFile":"522a892c-08a3-4633-b8ca-5698acbb3805.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"c6d1a2a5-5520-431f-a776-2e16862b2747","taskId":"T05-mechanical-codemod","mode":"write","engine":"cursor","model":"composer-2.5","effort":"default","repetition":2,"startedAt":"2026-10-13T08:44:47.242Z","finishedAt":"2026-10-13T08:45:53.625Z","wallClockSeconds":66.134863917,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"e6f69b1b9aec55f914554ed0379c1d37","jobState":"done","failureKind":null,"resolvedModel":"composer-2.5","resolvedEffort":null,"billingPool":"first-party","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":40732,"outputTokens":3119,"cacheReadTokens":17088,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":0,"verdict":"pass","infrastructureFailure":null,"resultFile":null,"diffFile":"c6d1a2a5-5520-431f-a776-2e16862b2747.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"9bfae723-5a9c-4ac2-a38b-ef2787ef2846","taskId":"T05-mechanical-codemod","mode":"write","engine":"cursor","model":"composer-2.5","effort":"default","repetition":3,"startedAt":"2026-10-13T08:44:47.242Z","finishedAt":"2026-10-13T08:45:55.566Z","wallClockSeconds":68.075884667,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"3bdeda31ac29773ed9b6c73e2a122c3f","jobState":"done","failureKind":null,"resolvedModel":"composer-2.5","resolvedEffort":null,"billingPool":"first-party","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":39681,"outputTokens":2803,"cacheReadTokens":17088,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":0,"verdict":"pass","infrastructureFailure":null,"resultFile":null,"diffFile":"9bfae723-5a9c-4ac2-a38b-ef2787ef2846.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"cfc59682-3cf5-4d11-93ed-c30442982945","taskId":"T09-style-fidelity","mode":"write","engine":"cursor","model":"composer-2.5","effort":"default","repetition":2,"startedAt":"2026-10-13T08:44:47.243Z","finishedAt":"2026-10-13T08:46:02.615Z","wallClockSeconds":75.102112166,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"217c7eb51ab75ea1c8cbcd7cddc738c3","jobState":"done","failureKind":null,"resolvedModel":"composer-2.5","resolvedEffort":null,"billingPool":"first-party","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":43772,"outputTokens":3527,"cacheReadTokens":19936,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":0,"verdict":"pass","infrastructureFailure":null,"resultFile":null,"diffFile":"cfc59682-3cf5-4d11-93ed-c30442982945.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"4bfd6028-7871-493b-8f6c-82e016889986","taskId":"T09-style-fidelity","mode":"write","engine":"cursor","model":"composer-2.5","effort":"default","repetition":1,"startedAt":"2026-10-13T08:44:47.242Z","finishedAt":"2026-10-13T08:46:05.648Z","wallClockSeconds":78.142481541,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"c594ee914e3ddd8e9263368d48ba9a84","jobState":"done","failureKind":null,"resolvedModel":"composer-2.5","resolvedEffort":null,"billingPool":"first-party","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":8330,"outputTokens":3743,"cacheReadTokens":36097,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":0,"verdict":"pass","infrastructureFailure":null,"resultFile":null,"diffFile":"4bfd6028-7871-493b-8f6c-82e016889986.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"9ee090ce-f761-4602-b715-b762067b656f","taskId":"T06-plan-shaped-fixes","mode":"write","engine":"cursor","model":"composer-2.5-fast","effort":"default","repetition":2,"startedAt":"2026-10-13T08:45:48.961Z","finishedAt":"2026-10-13T08:46:45.463Z","wallClockSeconds":56.280818708,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"3281f027431279043f78a24ee4296ec3","jobState":"done","failureKind":null,"resolvedModel":"composer-2.5-fast","resolvedEffort":null,"billingPool":"first-party","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":22235,"outputTokens":1870,"cacheReadTokens":19328,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":1,"verdict":"fail","infrastructureFailure":null,"resultFile":null,"diffFile":"9ee090ce-f761-4602-b715-b762067b656f.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"1f7dab0e-d579-442c-a00d-e168ba97833d","taskId":"T06-plan-shaped-fixes","mode":"write","engine":"cursor","model":"composer-2.5-fast","effort":"default","repetition":3,"startedAt":"2026-10-13T08:45:50.657Z","finishedAt":"2026-10-13T08:46:46.592Z","wallClockSeconds":55.696566417,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"c8e73282aac3152b4757116c01800c4a","jobState":"done","failureKind":null,"resolvedModel":"composer-2.5-fast","resolvedEffort":null,"billingPool":"first-party","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":28531,"outputTokens":1946,"cacheReadTokens":22592,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":1,"verdict":"fail","infrastructureFailure":null,"resultFile":null,"diffFile":"1f7dab0e-d579-442c-a00d-e168ba97833d.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"cf4c7160-48e5-4ee5-b8f7-22b593df89aa","taskId":"T06-plan-shaped-fixes","mode":"write","engine":"cursor","model":"composer-2.5-fast","effort":"default","repetition":1,"startedAt":"2026-10-13T08:45:48.885Z","finishedAt":"2026-10-13T08:46:46.732Z","wallClockSeconds":57.644951291,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"4aecb564d02bc0e5eea31a235aa6c4ab","jobState":"done","failureKind":null,"resolvedModel":"composer-2.5-fast","resolvedEffort":null,"billingPool":"first-party","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":35889,"outputTokens":1950,"cacheReadTokens":17920,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":1,"verdict":"fail","infrastructureFailure":null,"resultFile":null,"diffFile":"cf4c7160-48e5-4ee5-b8f7-22b593df89aa.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"36acb6c7-ee09-414f-aa4f-591568773567","taskId":"T05-mechanical-codemod","mode":"write","engine":"cursor","model":"composer-2.5-fast","effort":"default","repetition":2,"startedAt":"2026-10-13T08:45:48.015Z","finishedAt":"2026-10-13T08:46:49.714Z","wallClockSeconds":61.485014,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"3dc6a3c6315f3b8e6cf85847f4ac8713","jobState":"done","failureKind":null,"resolvedModel":"composer-2.5-fast","resolvedEffort":null,"billingPool":"first-party","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":45411,"outputTokens":3228,"cacheReadTokens":21600,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":0,"verdict":"pass","infrastructureFailure":null,"resultFile":null,"diffFile":"36acb6c7-ee09-414f-aa4f-591568773567.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"27b121c9-a545-4768-935d-0adabb57a1b7","taskId":"T09-style-fidelity","mode":"write","engine":"cursor","model":"composer-2.5-fast","effort":"default","repetition":1,"startedAt":"2026-10-13T08:45:53.675Z","finishedAt":"2026-10-13T08:46:53.407Z","wallClockSeconds":59.50556025,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"5ef86c645bacdaf5ee4dbe7e984f9d57","jobState":"done","failureKind":null,"resolvedModel":"composer-2.5-fast","resolvedEffort":null,"billingPool":"first-party","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":33181,"outputTokens":3136,"cacheReadTokens":19584,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":0,"verdict":"pass","infrastructureFailure":null,"resultFile":null,"diffFile":"27b121c9-a545-4768-935d-0adabb57a1b7.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"2f6dda9b-93af-40ea-85bf-a43d8d3a0481","taskId":"T09-style-fidelity","mode":"write","engine":"cursor","model":"composer-2.5-fast","effort":"default","repetition":3,"startedAt":"2026-10-13T08:46:02.660Z","finishedAt":"2026-10-13T08:46:59.878Z","wallClockSeconds":56.981549,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"b746a1e657d9c4cba0445e46e5306df3","jobState":"done","failureKind":null,"resolvedModel":"composer-2.5-fast","resolvedEffort":null,"billingPool":"first-party","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":37538,"outputTokens":3148,"cacheReadTokens":5152,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":0,"verdict":"pass","infrastructureFailure":null,"resultFile":null,"diffFile":"2f6dda9b-93af-40ea-85bf-a43d8d3a0481.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"327f58bf-352f-46ab-ac46-0360c28c362f","taskId":"T05-mechanical-codemod","mode":"write","engine":"cursor","model":"composer-2.5-fast","effort":"default","repetition":3,"startedAt":"2026-10-13T08:45:48.742Z","finishedAt":"2026-10-13T08:47:08.042Z","wallClockSeconds":79.088363625,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"80dc4e52cd7f4bf150ca001b54d21333","jobState":"done","failureKind":null,"resolvedModel":"composer-2.5-fast","resolvedEffort":null,"billingPool":"first-party","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":25240,"outputTokens":3058,"cacheReadTokens":31904,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":0,"verdict":"pass","infrastructureFailure":null,"resultFile":null,"diffFile":"327f58bf-352f-46ab-ac46-0360c28c362f.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"8101c870-4e4d-4521-97af-ba44f94199b5","taskId":"T09-style-fidelity","mode":"write","engine":"cursor","model":"composer-2.5-fast","effort":"default","repetition":2,"startedAt":"2026-10-13T08:45:55.611Z","finishedAt":"2026-10-13T08:47:24.415Z","wallClockSeconds":88.583938708,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"e02fa1196b8c35b785011f3118dfadc8","jobState":"done","failureKind":null,"resolvedModel":"composer-2.5-fast","resolvedEffort":null,"billingPool":"first-party","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":37882,"outputTokens":4840,"cacheReadTokens":31552,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":0,"verdict":"pass","infrastructureFailure":null,"resultFile":null,"diffFile":"8101c870-4e4d-4521-97af-ba44f94199b5.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"b264fb1c-5e16-404d-94a4-50d25574247f","taskId":"T05-mechanical-codemod","mode":"write","engine":"codex","model":"gpt-6-luna","effort":"xhigh","repetition":3,"startedAt":"2026-10-13T08:49:27.599Z","finishedAt":"2026-10-13T08:50:56.038Z","wallClockSeconds":88.184068833,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/codex/0.0.63/scripts/codex-companion.mjs","companionExit":0,"jobId":"695054924190a39e8e9bffff892e0529","jobState":"done","failureKind":null,"resolvedModel":"gpt-6-luna","resolvedEffort":"xhigh","billingPool":null,"cliVersion":"0.162.1","usage":{"inputTokens":103015,"cachedInputTokens":63232,"outputTokens":2753,"reasoningOutputTokens":683,"totalTokens":105768},"totalCostUsd":null,"verifyExit":0,"verdict":"pass","infrastructureFailure":null,"resultFile":null,"diffFile":"b264fb1c-5e16-404d-94a4-50d25574247f.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"ed1592e8-4c60-4bee-bcfd-84a45a5fca98","taskId":"T06-plan-shaped-fixes","mode":"write","engine":"codex","model":"gpt-6-luna","effort":"xhigh","repetition":2,"startedAt":"2026-10-13T08:49:27.598Z","finishedAt":"2026-10-13T08:51:05.863Z","wallClockSeconds":98.032333542,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/codex/0.0.63/scripts/codex-companion.mjs","companionExit":0,"jobId":"010d529e775c7c2083d17db4e5fe2e3a","jobState":"done","failureKind":null,"resolvedModel":"gpt-6-luna","resolvedEffort":"xhigh","billingPool":null,"cliVersion":"0.162.1","usage":{"inputTokens":104254,"cachedInputTokens":66048,"outputTokens":3921,"reasoningOutputTokens":2056,"totalTokens":108175},"totalCostUsd":null,"verifyExit":0,"verdict":"pass","infrastructureFailure":null,"resultFile":null,"diffFile":"ed1592e8-4c60-4bee-bcfd-84a45a5fca98.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"1f5b8bba-15ee-44b5-b484-f98c7b2fd6f0","taskId":"T06-plan-shaped-fixes","mode":"write","engine":"codex","model":"gpt-6-luna","effort":"xhigh","repetition":3,"startedAt":"2026-10-13T08:49:27.599Z","finishedAt":"2026-10-13T08:51:12.625Z","wallClockSeconds":104.770881625,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/codex/0.0.63/scripts/codex-companion.mjs","companionExit":0,"jobId":"6d686d42f4f13d8f74ef7643c94d9377","jobState":"done","failureKind":null,"resolvedModel":"gpt-6-luna","resolvedEffort":"xhigh","billingPool":null,"cliVersion":"0.162.1","usage":{"inputTokens":137847,"cachedInputTokens":100608,"outputTokens":3664,"reasoningOutputTokens":1876,"totalTokens":141511},"totalCostUsd":null,"verifyExit":0,"verdict":"pass","infrastructureFailure":null,"resultFile":null,"diffFile":"1f5b8bba-15ee-44b5-b484-f98c7b2fd6f0.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"bd968e93-12d2-4e12-96d9-28cd769d7cdd","taskId":"T05-mechanical-codemod","mode":"write","engine":"codex","model":"gpt-6-luna","effort":"xhigh","repetition":2,"startedAt":"2026-10-13T08:49:27.599Z","finishedAt":"2026-10-13T08:51:14.204Z","wallClockSeconds":106.352098375,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/codex/0.0.63/scripts/codex-companion.mjs","companionExit":0,"jobId":"00ae64485dff2037f83251f1e2164815","jobState":"done","failureKind":null,"resolvedModel":"gpt-6-luna","resolvedEffort":"xhigh","billingPool":null,"cliVersion":"0.162.1","usage":{"inputTokens":117696,"cachedInputTokens":95488,"outputTokens":2963,"reasoningOutputTokens":838,"totalTokens":120659},"totalCostUsd":null,"verifyExit":0,"verdict":"pass","infrastructureFailure":null,"resultFile":null,"diffFile":"bd968e93-12d2-4e12-96d9-28cd769d7cdd.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"ebd380bd-02b0-4a30-b179-5f7961dff1ea","taskId":"T06-plan-shaped-fixes","mode":"write","engine":"codex","model":"gpt-6-luna","effort":"xhigh","repetition":1,"startedAt":"2026-10-13T08:49:27.599Z","finishedAt":"2026-10-13T08:51:20.169Z","wallClockSeconds":112.342713083,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/codex/0.0.63/scripts/codex-companion.mjs","companionExit":0,"jobId":"034b52826c332a0ecc73ba44ad9615cf","jobState":"done","failureKind":null,"resolvedModel":"gpt-6-luna","resolvedEffort":"xhigh","billingPool":null,"cliVersion":"0.162.1","usage":{"inputTokens":113135,"cachedInputTokens":89600,"outputTokens":4174,"reasoningOutputTokens":2366,"totalTokens":117309},"totalCostUsd":null,"verifyExit":0,"verdict":"pass","infrastructureFailure":null,"resultFile":null,"diffFile":"ebd380bd-02b0-4a30-b179-5f7961dff1ea.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"05055fb6-c05d-4fa3-bd36-93be752a3724","taskId":"T09-style-fidelity","mode":"write","engine":"codex","model":"gpt-6-luna","effort":"xhigh","repetition":3,"startedAt":"2026-10-13T08:49:27.599Z","finishedAt":"2026-10-13T08:51:29.175Z","wallClockSeconds":121.224245334,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/codex/0.0.63/scripts/codex-companion.mjs","companionExit":0,"jobId":"a9fe9017ab9abc9324adee2c7ddc585b","jobState":"done","failureKind":null,"resolvedModel":"gpt-6-luna","resolvedEffort":"xhigh","billingPool":null,"cliVersion":"0.162.1","usage":{"inputTokens":103743,"cachedInputTokens":79616,"outputTokens":5302,"reasoningOutputTokens":3945,"totalTokens":109045},"totalCostUsd":null,"verifyExit":0,"verdict":"pass","infrastructureFailure":null,"resultFile":null,"diffFile":"05055fb6-c05d-4fa3-bd36-93be752a3724.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"bd7ca6a8-de22-4253-a776-e5a5b6668873","taskId":"T09-style-fidelity","mode":"write","engine":"codex","model":"gpt-6-luna","effort":"xhigh","repetition":2,"startedAt":"2026-10-13T08:49:27.599Z","finishedAt":"2026-10-13T08:51:38.389Z","wallClockSeconds":130.530483041,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/codex/0.0.63/scripts/codex-companion.mjs","companionExit":0,"jobId":"a88cb98f4f929c11c8fda5b36dcfa63b","jobState":"done","failureKind":null,"resolvedModel":"gpt-6-luna","resolvedEffort":"xhigh","billingPool":null,"cliVersion":"0.162.1","usage":{"inputTokens":129270,"cachedInputTokens":104704,"outputTokens":6715,"reasoningOutputTokens":4803,"totalTokens":135985},"totalCostUsd":null,"verifyExit":0,"verdict":"pass","infrastructureFailure":null,"resultFile":null,"diffFile":"bd7ca6a8-de22-4253-a776-e5a5b6668873.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"09d90cc6-daa5-4a99-8f21-572d67e94522","taskId":"T05-mechanical-codemod","mode":"write","engine":"codex","model":"gpt-6-luna","effort":"xhigh","repetition":1,"startedAt":"2026-10-13T08:49:27.599Z","finishedAt":"2026-10-13T08:51:39.079Z","wallClockSeconds":131.227025416,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/codex/0.0.63/scripts/codex-companion.mjs","companionExit":0,"jobId":"a0827f385980cac250824d5c0aeef66e","jobState":"done","failureKind":null,"resolvedModel":"gpt-6-luna","resolvedEffort":"xhigh","billingPool":null,"cliVersion":"0.162.1","usage":{"inputTokens":145720,"cachedInputTokens":101376,"outputTokens":5498,"reasoningOutputTokens":1612,"totalTokens":151218},"totalCostUsd":null,"verifyExit":0,"verdict":"pass","infrastructureFailure":null,"resultFile":null,"diffFile":"09d90cc6-daa5-4a99-8f21-572d67e94522.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"8fe18b63-bffd-4913-8021-6b185a466f92","taskId":"T09-style-fidelity","mode":"write","engine":"codex","model":"gpt-6-luna","effort":"xhigh","repetition":1,"startedAt":"2026-10-13T08:49:27.599Z","finishedAt":"2026-10-13T08:51:51.058Z","wallClockSeconds":143.200935833,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/codex/0.0.63/scripts/codex-companion.mjs","companionExit":0,"jobId":"0919c364274e2da090d67f18d5a3d77d","jobState":"done","failureKind":null,"resolvedModel":"gpt-6-luna","resolvedEffort":"xhigh","billingPool":null,"cliVersion":"0.162.1","usage":{"inputTokens":160414,"cachedInputTokens":121856,"outputTokens":6493,"reasoningOutputTokens":4641,"totalTokens":166907},"totalCostUsd":null,"verifyExit":0,"verdict":"pass","infrastructureFailure":null,"resultFile":null,"diffFile":"8fe18b63-bffd-4913-8021-6b185a466f92.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"628f29d7-f52e-47f4-9608-5a70c4581a47","taskId":"T04-checkable-research","mode":"consult","engine":"cursor","model":"muse-spark-1.3","effort":"xhigh","repetition":1,"startedAt":"2026-10-13T08:52:52.297Z","finishedAt":"2026-10-13T08:53:25.442Z","wallClockSeconds":32.824991791,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"5750ed9189e5bf4f2a19ede4f55b3028","jobState":"done","failureKind":null,"resolvedModel":"muse-spark-1.3-xhigh","resolvedEffort":null,"billingPool":"api","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":8661,"outputTokens":2088,"cacheReadTokens":8930,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":null,"verdict":"unscored","infrastructureFailure":null,"resultFile":"628f29d7-f52e-47f4-9608-5a70c4581a47.result.md","diffFile":null,"taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"c2cef0f5-7600-4401-8615-43ad558bf073","taskId":"T06-plan-shaped-fixes","mode":"consult","engine":"cursor","model":"muse-spark-1.3","effort":"xhigh","repetition":1,"startedAt":"2026-10-13T08:52:52.297Z","finishedAt":"2026-10-13T08:53:41.201Z","wallClockSeconds":48.585508458,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"d93c8e89286730f6b27ab37facec1908","jobState":"done","failureKind":null,"resolvedModel":"muse-spark-1.3-xhigh","resolvedEffort":null,"billingPool":"api","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":1842,"outputTokens":2820,"cacheReadTokens":6626,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":null,"verdict":"unscored","infrastructureFailure":null,"resultFile":"c2cef0f5-7600-4401-8615-43ad558bf073.result.md","diffFile":null,"taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"2fedd008-b589-44e5-bdd1-5bf8c141df70","taskId":"T02-negative-control","mode":"consult","engine":"cursor","model":"muse-spark-1.3","effort":"xhigh","repetition":1,"startedAt":"2026-10-13T08:52:52.309Z","finishedAt":"2026-10-13T08:53:42.354Z","wallClockSeconds":49.730949709,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"819a40529d04a9b534980a53d117d94f","jobState":"done","failureKind":null,"resolvedModel":"muse-spark-1.3-xhigh","resolvedEffort":null,"billingPool":"api","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":1830,"outputTokens":3598,"cacheReadTokens":10707,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":null,"verdict":"unscored","infrastructureFailure":null,"resultFile":"2fedd008-b589-44e5-bdd1-5bf8c141df70.result.md","diffFile":null,"taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"c916076b-9039-4a14-b7e6-da9604ec6d0e","taskId":"T01-seeded-bug","mode":"consult","engine":"cursor","model":"muse-spark-1.3","effort":"xhigh","repetition":1,"startedAt":"2026-10-13T08:52:52.308Z","finishedAt":"2026-10-13T08:54:00.935Z","wallClockSeconds":68.324831375,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"afb136ab85952adaaa45d3c13fb6bfe2","jobState":"done","failureKind":null,"resolvedModel":"muse-spark-1.3-xhigh","resolvedEffort":null,"billingPool":"api","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":9041,"outputTokens":4507,"cacheReadTokens":22084,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":null,"verdict":"unscored","infrastructureFailure":null,"resultFile":"c916076b-9039-4a14-b7e6-da9604ec6d0e.result.md","diffFile":null,"taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"34cd4219-fac1-40ff-b53a-e6ff19e13cd5","taskId":"T01-seeded-bug","mode":"consult","engine":"cursor","model":"kimi-k3","effort":"high","repetition":1,"startedAt":"2026-10-13T08:53:25.522Z","finishedAt":"2026-10-13T08:54:01.229Z","wallClockSeconds":35.630100667,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"16fa5cfcc343ef85b496395ef72cc700","jobState":"done","failureKind":null,"resolvedModel":"kimi-k3-high","resolvedEffort":null,"billingPool":"api","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":11769,"outputTokens":3059,"cacheReadTokens":0,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":null,"verdict":"unscored","infrastructureFailure":null,"resultFile":"34cd4219-fac1-40ff-b53a-e6ff19e13cd5.result.md","diffFile":null,"taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"91fba62f-39bc-492d-a987-d27407dd401d","taskId":"T04-checkable-research","mode":"consult","engine":"cursor","model":"gemini-3.8-flash","effort":"high","repetition":1,"startedAt":"2026-10-13T08:52:52.298Z","finishedAt":"2026-10-13T08:54:04.446Z","wallClockSeconds":71.830171,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"8b00bb9f6d3b98e1099029b3325594f2","jobState":"done","failureKind":null,"resolvedModel":"gemini-3.8-flash-high","resolvedEffort":null,"billingPool":"api","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":27623,"outputTokens":5838,"cacheReadTokens":19562,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":null,"verdict":"unscored","infrastructureFailure":null,"resultFile":"91fba62f-39bc-492d-a987-d27407dd401d.result.md","diffFile":null,"taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"822ae104-2933-4fa2-aca4-0f5dced6e856","taskId":"T06-plan-shaped-fixes","mode":"consult","engine":"cursor","model":"kimi-k3","effort":"high","repetition":1,"startedAt":"2026-10-13T08:53:41.267Z","finishedAt":"2026-10-13T08:54:08.705Z","wallClockSeconds":27.353460833,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"a96d444ac80b5fbad02f98a57ab42b65","jobState":"done","failureKind":null,"resolvedModel":"kimi-k3-high","resolvedEffort":null,"billingPool":"api","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":3849,"outputTokens":1187,"cacheReadTokens":3584,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":null,"verdict":"unscored","infrastructureFailure":null,"resultFile":"822ae104-2933-4fa2-aca4-0f5dced6e856.result.md","diffFile":null,"taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"7acc33e1-8e4b-4535-b28e-d807a2217364","taskId":"T04-checkable-research","mode":"consult","engine":"cursor","model":"kimi-k3","effort":"high","repetition":1,"startedAt":"2026-10-13T08:53:42.397Z","finishedAt":"2026-10-13T08:54:14.176Z","wallClockSeconds":31.700875834,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"b8f0890a7d5ec3b2fc33c6246284cb15","jobState":"done","failureKind":null,"resolvedModel":"kimi-k3-high","resolvedEffort":null,"billingPool":"api","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":22785,"outputTokens":1108,"cacheReadTokens":3840,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":null,"verdict":"unscored","infrastructureFailure":null,"resultFile":"7acc33e1-8e4b-4535-b28e-d807a2217364.result.md","diffFile":null,"taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"88a188a4-60d2-4013-ba45-fa90bfab5865","taskId":"T02-negative-control","mode":"consult","engine":"cursor","model":"kimi-k3","effort":"high","repetition":1,"startedAt":"2026-10-13T08:54:00.982Z","finishedAt":"2026-10-13T08:54:29.646Z","wallClockSeconds":28.51857575,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"0a2e8919d67967e1b186ef61b341984e","jobState":"done","failureKind":null,"resolvedModel":"kimi-k3-high","resolvedEffort":null,"billingPool":"api","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":11348,"outputTokens":2103,"cacheReadTokens":1344,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":null,"verdict":"unscored","infrastructureFailure":null,"resultFile":"88a188a4-60d2-4013-ba45-fa90bfab5865.result.md","diffFile":null,"taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"37cb5adf-e98d-4d2e-87a6-e7cbbb62e5ad","taskId":"T06-plan-shaped-fixes","mode":"consult","engine":"cursor","model":"glm-5.2","effort":"high","repetition":1,"startedAt":"2026-10-13T08:54:04.493Z","finishedAt":"2026-10-13T08:54:31.559Z","wallClockSeconds":26.986009875,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"f03cee800d89885a62f8f6223d883d8e","jobState":"done","failureKind":null,"resolvedModel":"glm-5.2-high","resolvedEffort":null,"billingPool":"api","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":2428,"outputTokens":1140,"cacheReadTokens":5088,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":null,"verdict":"unscored","infrastructureFailure":null,"resultFile":"37cb5adf-e98d-4d2e-87a6-e7cbbb62e5ad.result.md","diffFile":null,"taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"13c103d0-6823-4a99-b568-03ef7413f1e5","taskId":"T04-checkable-research","mode":"consult","engine":"cursor","model":"glm-5.2","effort":"high","repetition":1,"startedAt":"2026-10-13T08:54:08.749Z","finishedAt":"2026-10-13T08:54:44.724Z","wallClockSeconds":35.886023458,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"ae834d3714d6542c3637c8ecc4f94539","jobState":"done","failureKind":null,"resolvedModel":"glm-5.2-high","resolvedEffort":null,"billingPool":"api","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":4683,"outputTokens":1164,"cacheReadTokens":27712,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":null,"verdict":"unscored","infrastructureFailure":null,"resultFile":"13c103d0-6823-4a99-b568-03ef7413f1e5.result.md","diffFile":null,"taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"3f6dc466-3903-4d6e-bb3d-97f5bd884d49","taskId":"T01-seeded-bug","mode":"consult","engine":"cursor","model":"glm-5.2","effort":"high","repetition":1,"startedAt":"2026-10-13T08:54:01.282Z","finishedAt":"2026-10-13T08:54:45.445Z","wallClockSeconds":44.067076209,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"12c302009e618f80a3bf7ef90290197f","jobState":"done","failureKind":null,"resolvedModel":"glm-5.2-high","resolvedEffort":null,"billingPool":"api","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":2173,"outputTokens":1305,"cacheReadTokens":13760,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":null,"verdict":"unscored","infrastructureFailure":null,"resultFile":"3f6dc466-3903-4d6e-bb3d-97f5bd884d49.result.md","diffFile":null,"taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"5455dfba-d819-4a7d-933f-61e0fabce120","taskId":"T02-negative-control","mode":"consult","engine":"cursor","model":"glm-5.2","effort":"high","repetition":1,"startedAt":"2026-10-13T08:54:14.219Z","finishedAt":"2026-10-13T08:54:56.548Z","wallClockSeconds":42.257683875,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"92ae3f2bab7cb620c5120b64e0e4aece","jobState":"done","failureKind":null,"resolvedModel":"glm-5.2-high","resolvedEffort":null,"billingPool":"api","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":1670,"outputTokens":1245,"cacheReadTokens":13408,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":null,"verdict":"unscored","infrastructureFailure":null,"resultFile":"5455dfba-d819-4a7d-933f-61e0fabce120.result.md","diffFile":null,"taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"30dccbf3-35a6-4701-84c9-7420dacd8cb5","taskId":"T02-negative-control","mode":"consult","engine":"cursor","model":"gemini-3.8-flash","effort":"high","repetition":1,"startedAt":"2026-10-13T08:52:52.312Z","finishedAt":"2026-10-13T08:56:01.862Z","wallClockSeconds":189.244215,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"7d818cb43730e33d5ce21ce7c3a254b1","jobState":"done","failureKind":null,"resolvedModel":"gemini-3.8-flash-high","resolvedEffort":null,"billingPool":"api","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":38251,"outputTokens":319,"cacheReadTokens":0,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":null,"verdict":"unscored","infrastructureFailure":null,"resultFile":"30dccbf3-35a6-4701-84c9-7420dacd8cb5.result.md","diffFile":null,"taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"3ed1de8c-e3b9-4e08-be7b-f5f177670157","taskId":"T01-seeded-bug","mode":"consult","engine":"cursor","model":"gemini-3.8-flash","effort":"high","repetition":1,"startedAt":"2026-10-13T08:52:52.299Z","finishedAt":"2026-10-13T08:56:15.013Z","wallClockSeconds":202.389205917,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"c7c58c48c61f4bdc5ef3cbbfbaee479e","jobState":"done","failureKind":null,"resolvedModel":"gemini-3.8-flash-high","resolvedEffort":null,"billingPool":"api","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":36402,"outputTokens":25329,"cacheReadTokens":23473,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":null,"verdict":"unscored","infrastructureFailure":null,"resultFile":"3ed1de8c-e3b9-4e08-be7b-f5f177670157.result.md","diffFile":null,"taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"30311345-9d7c-4f8f-a58a-582bdfe637ae","taskId":"T06-plan-shaped-fixes","mode":"consult","engine":"cursor","model":"gemini-3.8-flash","effort":"high","repetition":1,"startedAt":"2026-10-13T08:52:52.309Z","finishedAt":"2026-10-13T08:56:42.645Z","wallClockSeconds":230.026121792,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/cursor/0.0.63/scripts/cursor-companion.mjs","companionExit":0,"jobId":"12e3add7d81c5ff625cb8021244a7bba","jobState":"done","failureKind":null,"resolvedModel":"gemini-3.8-flash-high","resolvedEffort":null,"billingPool":"api","cliVersion":"2026.10.01-e373342","usage":{"inputTokens":28300,"outputTokens":1243,"cacheReadTokens":0,"cacheWriteTokens":0},"totalCostUsd":null,"verifyExit":null,"verdict":"unscored","infrastructureFailure":null,"resultFile":"30311345-9d7c-4f8f-a58a-582bdfe637ae.result.md","diffFile":null,"taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} +{"schema":"fusion-bench-lane/v1","runId":"b0b18dee-2b44-48b7-8455-dd2deab1b18b","taskId":"T05-mechanical-codemod","mode":"write","engine":"grok","model":"grok-4.7-build-fast","effort":"low","repetition":1,"startedAt":"2026-10-13T09:01:41.879Z","finishedAt":"2026-10-13T09:01:42.196Z","wallClockSeconds":0.108909916,"companionPath":"~/.claude/plugins/cache/claude-code-fusion/grok/0.0.63/scripts/grok-companion.mjs","companionExit":1,"jobId":"3cc7ececb1ee62ffe8115f4925f0dd67","jobState":"error","failureKind":"sandbox","resolvedModel":"grok-4.7-build-fast","resolvedEffort":"low","billingPool":null,"cliVersion":"1.0.52","usage":null,"totalCostUsd":null,"verifyExit":1,"verdict":"fail","infrastructureFailure":null,"resultFile":null,"diffFile":"b0b18dee-2b44-48b7-8455-dd2deab1b18b.diff","taskManifestHash":"e0d1c68370ef0a682e23da0850d081e91365a964c95bf6386e67085114168967"} diff --git a/bench/results/lanes-2026-10-13/summary.md b/bench/results/lanes-2026-10-13/summary.md new file mode 100644 index 0000000..7cf99fe --- /dev/null +++ b/bench/results/lanes-2026-10-13/summary.md @@ -0,0 +1,43 @@ +# Lane snapshot 2026-10-13 + +One machine (macOS 27.0, Node 24.21), plugins at 0.0.63, codex-cli 0.162.1, grok 1.0.52, Cursor Agent CLI 2026.10.01-e373342, task manifest `e0d1c683`. Every attempt is in `runs.jsonl`; `env.json` pins the environment. Diffs and consult answers stay out of the tree as the methodology requires. + +## Burst lanes + +Write mode, three repetitions per lane and task, each lane on its routing default. Input tokens follow each engine's own accounting: Cursor reports cache reads separately, while Codex input includes cached tokens. + +| Task | Lane | Passed | Median wall clock | Median input tokens | Median output tokens | +|---|---|---:|---:|---:|---:| +| T05 mechanical codemod | composer-2.5 | 3 of 3 | 68 s | 39,681 | 3,022 | +| T05 mechanical codemod | composer-2.5-fast | 3 of 3 | 61 s | 34,886 | 3,058 | +| T05 mechanical codemod | gpt-6-luna xhigh | 3 of 3 | 106 s | 117,696 | 2,963 | +| T06 plan shaped fixes | composer-2.5 | 0 of 3 | 61 s | 21,387 | 1,521 | +| T06 plan shaped fixes | composer-2.5-fast | 0 of 3 | 56 s | 28,531 | 1,946 | +| T06 plan shaped fixes | gpt-6-luna xhigh | 3 of 3 | 105 s | 113,135 | 3,921 | +| T09 style fidelity | composer-2.5 | 3 of 3 | 75 s | 38,176 | 3,527 | +| T09 style fidelity | composer-2.5-fast | 3 of 3 | 60 s | 37,538 | 3,148 | +| T09 style fidelity | gpt-6-luna xhigh | 3 of 3 | 131 s | 129,270 | 6,493 | + +Every Composer T06 attempt fixed all three packages but parsed CSV quoting only as far as the brief spells it out: a doubled double quote inside a quoted field stays doubled, which the hidden test rejects. Every luna attempt handled it. Composer finished in 46 to 64 percent of luna's wall clock on the tasks it passed. Across nine attempts each, composer-2.5 used 232,207 input and 23,826 output tokens, composer-2.5-fast 300,793 and 25,869, both from Cursor's first-party pool at $0.50 and $2.50 per million tokens for composer-2.5 against $3 and $15 for the fast variant. Luna used 1,115,094 input tokens (822,528 cached) and 41,483 output tokens at 2.5, 0.25, and 12.5 Codex credits per million. + +The grok burst arm (grok-4.7-build-fast at low) has no measurement. Its one attempt failed as `sandbox` before spawn because OrbStack's `~/.orbstack/run/docker.sock` exists on this host, the same refusal both fusion hosts hit, and the account's Grok Build balance is exhausted (HTTP 402), so neither host can run a grok write today. + +## Diversity seats + +Consult mode, one repetition per seat and brief, scored by the orchestrator against each brief's key. The lru brief has two real defects: the seeded eviction end, and a latent double removal that the T01 reference solution shares (`#dropKey` unlinks a node its caller already detached, which empties the list; with maxSize 2, inserting a, b, c, d leaves b and d). The blind review has three; the research brief has three values; the clean review has none, so every reported defect there is a false claim. + +| Seat | lru (2) | Blind review (3) | Research (3) | Clean review | False claims | Evidence slips | Wall clock | +|---|---:|---:|---:|---|---:|---|---:| +| fusion:deep-reasoner (Fable) | 2 | 3 | 3 | none found, one hedged overflow note | 0 | one predicted output in its own repro was a trace error | 65 to 221 s | +| muse-spark-1.3 xhigh | 2 | 3 | 3 | none found | 0 | the CSV fix is only sketched | 32 to 68 s | +| kimi-k3 high | 2 | 3 | 3 | none found | 0 | none | 27 to 35 s | +| gemini-3.8-flash high | 2 | 3 | 3 | one false defect (normalize with a reversed range) | 1 | none | 71 to 230 s | +| glm-5.2 high | 1 | 3 | 3 | none found | 0 | two line numbers one off, and it asserted the lru code had no other defect | 26 to 44 s | + +Deep-reasoner gave the strongest evidence: pointer level traces, a steelman of the opposite list convention, and scope notes kept apart from defects. muse-spark-1.3 and kimi-k3 reached the same findings, the latent defect included, with no false claims. The research brief separated nobody. + +## Routing reading + +- Burst: no promotion. Composer cannot be ranked against grok burst without a grok measurement, and it trails the Codex volume tier on plan shaped fixes (0 of 6 against 3 of 3). It stays co-equal with grok burst on the admission ladder, and on hosts where grok writes are refused it is the only burst lane besides Codex. composer-2.5-fast earns no seat over composer-2.5: same verdicts, seconds faster, six times the token price. +- Diversity: muse-spark-1.3 stays the default seat, and kimi-k3 is an equal second whenever Composer is not seated on the same question, because the two share the Kimi lineage. gemini-3.8-flash and glm-5.2 rank below them on this snapshot. +- Codex flight fit: five scoped read only source audits ran on gpt-6.1-sol at xhigh the same day; one finished in 405 s and four exceeded the 570 s foreground flight and finished through the wind down resume after 793 to 867 s, so the quick lane default stays gpt-6-sol. From b8b6e478449798673ed88ecd333590a72c52fe87 Mon Sep 17 00:00:00 2001 From: Harry Yep Date: Tue, 13 Oct 2026 17:25:49 +0800 Subject: [PATCH 5/8] feat(fusion): pin trivial-worker to claude-haiku-5-5 --- plugins/fusion/agents/trivial-worker.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/fusion/agents/trivial-worker.md b/plugins/fusion/agents/trivial-worker.md index a3a79c4..7791451 100644 --- a/plugins/fusion/agents/trivial-worker.md +++ b/plugins/fusion/agents/trivial-worker.md @@ -1,7 +1,7 @@ --- name: trivial-worker description: Fallback tier only for exact, low risk, tiny packages when no eligible peer lane is available or Claude-only tools or privacy are required. Cheapest tier worker pinned to true Haiku for trivial single file edits, renames, small doc fixes, log digestion, and short mechanical checks where speed and cost matter more than depth. Change briefs include a verification command; analysis, drafting, and research briefs include explicit acceptance criteria. Anything ambiguous returns to the orchestrator for resolution instead. -model: claude-haiku-4-5 +model: claude-haiku-5-5 effort: low maxTurns: 32 background: false From 57b7b887e7f903903145d277092c084aadf4b92d Mon Sep 17 00:00:00 2001 From: Harry Yep Date: Tue, 13 Oct 2026 17:25:49 +0800 Subject: [PATCH 6/8] docs: correct the grok trust boundary and re-score the cursor rows from the lane snapshot --- README.md | 12 ++++++------ SECURITY.md | 4 ++-- docs/grok-contract.md | 10 +++++----- plugins/fusion/rules-manifest.json | 1 + plugins/fusion/rules/orchestration.md | 14 +++++++------- plugins/fusion/rules/troubleshooting.md | 4 ++-- plugins/fusion/skills/doctor/SKILL.md | 8 ++++---- tests/rules-sync.test.mjs | 8 +++++--- 8 files changed, 32 insertions(+), 29 deletions(-) diff --git a/README.md b/README.md index 78f268d..7e0076e 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ Direct `/codex:*` and `/grok:*` commands are user selected engine escape hatches ## Quick start -Prerequisites: Claude Code 2.1.198 or later and Node.js 22 or later (tested on Claude Code 2.1.220 and Node 22; the floors are not enforced), git, macOS or Linux, and the peer CLIs you want installed and authenticated on PATH (contract verified against codex-cli 0.157.1, grok 1.0.41, and Cursor Agent CLI 2026.10.01-e373342; the Codex companion enforces a tested minimum version at preflight and fails with the setup kind below it, while the Grok companion refuses builds below 1.0.14 at preflight and `/grok:setup` reports builds outside its tested interval `[1.0.14, 1.0.31)`). Verify them with `codex --version` and `grok --version`. Claude workers and peer wrapper Agents use foreground delivery by default. The main Claude session remains the control plane and must collect or explicitly cancel any Agent that the runtime still launches asynchronously. Without one peer, Fusion still runs and routes eligible work to the remaining lanes. +Prerequisites: Claude Code 2.1.198 or later and Node.js 22 or later (tested on Claude Code 2.1.220 and Node 22; the floors are not enforced), git, macOS or Linux, and the peer CLIs you want installed and authenticated on PATH (contract verified against codex-cli 0.162.1, grok 1.0.41, and Cursor Agent CLI 2026.10.01-e373342; the Codex companion enforces a tested minimum version at preflight and fails with the setup kind below it, while the Grok companion refuses builds below 1.0.14 at preflight and `/grok:setup` reports builds outside its tested interval `[1.0.14, 1.0.31)`). Verify them with `codex --version` and `grok --version`. Claude workers and peer wrapper Agents use foreground delivery by default. The main Claude session remains the control plane and must collect or explicitly cancel any Agent that the runtime still launches asynchronously. Without one peer, Fusion still runs and routes eligible work to the remaining lanes. If `codex@openai-codex` is installed, migrate before installing this marketplace's Codex plugin. In a session where the old plugin is still enabled, run `/codex:status --all`, use the old `/codex:result --wait` or `/codex:cancel ` for every authoritative running job, confirm that no result you need remains uncollected, then exit that session. Disable and uninstall the old plugin while preserving its data: @@ -79,7 +79,7 @@ Grok consult, write, and review all pin `--sandbox strict`. Consult overrides in Before prompt delivery, the companion requires a new `ProfileApplied` event in the incremental delta of the shared `~/.grok/sessions/sandbox-events.jsonl` log (the home-root location that 1.0.13 and earlier wrote is unsupported) that contains this run's private temporary path and matches the canonical workspace, strict profile, enforced state, and requested `restrict_network: true` configuration. Upstream appends there with no rotation or size cap. Upstream records have no run id or pid, so the unique private path correlates positive evidence; unrelated `ProfileApplied` and `ApplyFailed` records are not attributed, and failure records are only auxiliary diagnostics. The matching event proves profile and configuration application, not actual platform network isolation. A matching warning on the owned stderr stream, handshake timeout, missing or malformed matching evidence, shared-log disappearance or rotation, or any matching field mismatch terminates the process and fails closed; stderr cannot prove successful enforcement. Forced builder tracing starts only after Grok consumes stdin, so it is not a pre-prompt or pre-side-effect attestation. Fallback or unmatched warnings trigger early termination, and a nominally successful close is rejected without positive allowlist applied evidence. -The child sets all eighteen `GROK_CLAUDE_*_ENABLED`, `GROK_CURSOR_*_ENABLED`, and `GROK_CODEX_*_ENABLED` bridge variables to false; upstream currently consumes the six Claude and six Cursor cells plus the Codex sessions cell and reserves the other five Codex cells, and the child also pins `GROK_MANAGED_MCPS_ENABLED=false` because that variable has highest precedence over `[managed_mcps]` in `~/.grok/config.toml` and remote settings. It disables subagents, removes inherited Claude session and plugin variables plus `_GROK_CLAUDE_MARKER_OVERRIDE`, scrubs unrelated secret-bearing variables while preserving required xAI authentication, and disables automatic updates with `--no-auto-update`. Cross-session memory is force-disabled for every run except an ordinary task with explicit `--memory`; review and stop gate always keep it off. When enabled, upstream may inject relevant Grok memory into the model context sent to xAI. Upstream automatic saving usually requires at least three real user prompts in the same resumed session, so one Fusion task usually only reads existing memory and does not guarantee that new memory is saved. The write tool and deny rules reduce exposure but are not complete confinement. Command-pattern denies for common direct grok, claude, and codex calls can be bypassed with absolute paths, aliases or functions, and indirect scripts; a hard boundary requires removing `run_terminal_cmd` or applying an OS-level executable or network policy. Run write delegations on a clean branch or an orchestrator-created disposable worktree and launch Grok with that worktree as its actual cwd. Prompts reach headless Grok through `/dev/stdin`, and stdout is parsed from a 0600 file that is unlinked immediately after open. The capability preflight checks the universal safety flags and the exact consult, write, review, no-web, and ordinary-wait surfaces before each applicable run; `/grok:setup` reports the complete surface. Grok splits soft update floors from hard startup requirements (introduced in 0.2.112, verified through 1.0.41); the hard layer exits 1 before the headless turn regardless of `--no-auto-update` and the companion classifies that death as `setup`. The companion's own tested interval is `[1.0.14, 1.0.31)`: below it managed runs fail closed as `setup`, at or past it `/grok:setup` reports the build as newer than the tested interval and a verification pass is advised; job records carry `grokVersion` so a breaker advisory can name the build that failed. `--web` deliberately uses Grok's built in network tools and may transmit inspected content to xAI. Continuity defaults to manual; optional `claude-session` affinity applies only to compatible ordinary direct tasks. Fusion routed briefs do not receive automatic affinity and stay fresh unless explicitly resumed. The optional stop gate runs from the Stop hook, not SessionEnd, and parses its first nonempty reply line: `ALLOW` allows the stop, while `BLOCK: ` blocks it. A `BLOCK` after a preamble and every infrastructure failure fail open. SessionEnd attempts verified process cleanup and removes unused raw transports, but it does not delete the terminal companion ledger. Grok permission matcher notes are in [docs/grok-contract.md](docs/grok-contract.md), and the shared outcome rules are in [docs/companion-contract.md](docs/companion-contract.md). +The child sets all eighteen `GROK_CLAUDE_*_ENABLED`, `GROK_CURSOR_*_ENABLED`, and `GROK_CODEX_*_ENABLED` bridge variables to false; upstream currently consumes the six Claude and six Cursor cells plus the Codex sessions cell and reserves the other five Codex cells, and the child also pins `GROK_MANAGED_MCPS_ENABLED=false` because that variable has highest precedence over `[managed_mcps]` in `~/.grok/config.toml` and remote settings. The bridge variables do not cover Claude Code plugins: Grok enables the plugins that `~/.claude/settings.json` enables unless `~/.grok/config.toml` sets `[claude_compat] imported = true` or lists them in `[plugins] disabled`, and their hooks and MCP servers can start inside managed runs. It disables subagents and the per turn summary model call, removes inherited Claude session and plugin variables plus `_GROK_CLAUDE_MARKER_OVERRIDE`, scrubs unrelated secret-bearing variables while preserving required xAI authentication, and disables automatic updates with `--no-auto-update`. Cross-session memory is force-disabled for every run except an ordinary task with explicit `--memory`; review and stop gate always keep it off. When enabled, upstream may inject relevant Grok memory into the model context sent to xAI. Upstream automatic saving usually requires at least three real user prompts in the same resumed session, so one Fusion task usually only reads existing memory and does not guarantee that new memory is saved. The write tool and deny rules reduce exposure but are not complete confinement. Command-pattern denies for common direct grok, claude, and codex calls can be bypassed with absolute paths, aliases or functions, and indirect scripts; a hard boundary requires removing `run_terminal_cmd` or applying an OS-level executable or network policy. Run write delegations on a clean branch or an orchestrator-created disposable worktree and launch Grok with that worktree as its actual cwd. Prompts reach headless Grok through `/dev/stdin`, and stdout is parsed from a 0600 file that is unlinked immediately after open. The capability preflight checks the universal safety flags and the exact consult, write, review, no-web, and ordinary-wait surfaces before each applicable run; `/grok:setup` reports the complete surface. Grok splits soft update floors from hard startup requirements (introduced in 0.2.112, verified through 1.0.41); the hard layer exits 1 before the headless turn regardless of `--no-auto-update` and the companion classifies that death as `setup`. The companion's own tested interval is `[1.0.14, 1.0.42)`: below it managed runs fail closed as `setup`, at or past it `/grok:setup` reports the build as newer than the tested interval and a verification pass is advised; job records carry `grokVersion` so a breaker advisory can name the build that failed. `--web` deliberately uses Grok's built in network tools and may transmit inspected content to xAI. Continuity defaults to manual; optional `claude-session` affinity applies only to compatible ordinary direct tasks. Fusion routed briefs do not receive automatic affinity and stay fresh unless explicitly resumed. The optional stop gate runs from the Stop hook, not SessionEnd, and parses its first nonempty reply line: `ALLOW` allows the stop, while `BLOCK: ` blocks it. A `BLOCK` after a preamble and every infrastructure failure fail open. SessionEnd attempts verified process cleanup and removes unused raw transports, but it does not delete the terminal companion ledger. Grok permission matcher notes are in [docs/grok-contract.md](docs/grok-contract.md), and the shared outcome rules are in [docs/companion-contract.md](docs/companion-contract.md). Upstream strict has broader read and write surfaces than the workspace and adapter-private run directory. Linux `system_read` includes `/var` and `/tmp`; macOS includes `/private` and the entire `~/Library`, so shared temporary content and substantial user application configuration and caches may be readable. Its write roots are the workspace and `~/.grok/sessions` (since 1.0.14 the rest of the Grok home is readable but not writable), always include `/tmp` and `/var/tmp`, and on macOS also include `/private/tmp`, `/private/var/tmp`, and all of `/private/var/folders`. The run's unique private `TMPDIR` is the adapter's preferred and handshake-verified temporary path, not the only writable temporary surface. The sandbox permits consult `read_file` to reach `~/.grok/auth.json`, config, and sessions because strict keeps the Grok home readable, and write terminal commands can receive preserved xAI authentication variables. The companion adds best-effort Read denies for `auth.json`, `mcp_credentials.json`, `config.toml`, `sessions`, `memory`, `logs`, and `debug` under the Grok home through absolute paths and standard `**/.grok` patterns, but raw path variants, symbolic links, and shell or indirect scripts can bypass those tool-level rules. Those Read denies govern model tool calls only. They do not block upstream's internal first-turn memory search and injection when the user explicitly supplies `--memory`. Environment scrubbing and path-pattern denies do not isolate credentials or memory content. Model-facing MCP and meta-tool denies do not prove that native MCP servers, plugins, or hooks configured under `~/.grok` did not start during agent construction or had no side effects; those bridge variables do not disable native Grok configuration. Narrower exposure requires a dedicated sandbox profile, an isolated Grok home, an authentication broker, or upstream path-level authorization. @@ -95,19 +95,19 @@ The Cursor companion runs every job with `HOME` set to a private home that it bu ## Model roles -Roles bind to alias tiers, not to specific models, so a same tier release (Sonnet 5 under `sonnet`, a new Opus generation) needs zero configuration, and `best[1m]` floats the orchestrator to future Fable releases. `opus` resolves to Opus 5.5 since Claude Code 2.1.280, and Opus 5.5 holds the orchestrator seat at parity with Fable 5.1 on published evidence; Opus 5 and older remain a temporary fallback to reverse with /model, not a lateral move. The one deliberate exception is `fusion:trivial-worker`, pinned to the full ID `claude-haiku-4-5` for machines where ANTHROPIC_DEFAULT_HAIKU_MODEL remaps the haiku alias; bump it by hand when a newer cheap tier ships. It is a fallback for trivial work when no eligible peer lane remains, or when Claude-only tools or privacy are required, not the default for single file tasks. Fusion-routed peer dispatches carry the routing table's lane defaults (`/fusion:config set-default`), so a CLI's interactive default model or effort never leaks into delegated runs; direct peer commands still use each CLI's own config (`~/.grok/config.toml`, `~/.codex/config.toml`). Alias semantics are Claude Code behavior as observed on 2.1.x, and nothing here sets your main model for you: pick it yourself, for example `/model best`. `/fusion:doctor` audits all of this. +Roles bind to alias tiers, not to specific models, so a same tier release (Sonnet 5 under `sonnet`, a new Opus generation) needs zero configuration, and `best[1m]` floats the orchestrator to future Fable releases. `opus` resolves to Opus 5.5 since Claude Code 2.1.280, and Opus 5.5 holds the orchestrator seat at parity with Fable 5.1 on published evidence; Opus 5 and older remain a temporary fallback to reverse with /model, not a lateral move. The one deliberate exception is `fusion:trivial-worker`, pinned to the full ID `claude-haiku-5-5` for machines where ANTHROPIC_DEFAULT_HAIKU_MODEL remaps the haiku alias; bump it by hand when a newer cheap tier ships. It is a fallback for trivial work when no eligible peer lane remains, or when Claude-only tools or privacy are required, not the default for single file tasks. Fusion-routed peer dispatches carry the routing table's lane defaults (`/fusion:config set-default`), so a CLI's interactive default model or effort never leaks into delegated runs; direct peer commands still use each CLI's own config (`~/.grok/config.toml`, `~/.codex/config.toml`). Alias semantics are Claude Code behavior as observed on 2.1.x, and nothing here sets your main model for you: pick it yourself, for example `/model best`. `/fusion:doctor` audits all of this. ## Benchmark The plugin's effectiveness claims are benchmarked, not asserted. The claims and the full protocol live in [bench/METHODOLOGY.md](bench/METHODOLOGY.md): C1 quota displacement (routing bills fewer tokens to the Claude subscription than a vanilla session at equal task success, split into a Claude tiers only arm, a peer offload arm, and an advisor profile arm where the main session runs on Sonnet and consults stronger tiers on demand), C2 wall clock compression on plan shaped tasks versus a protocol enforced sequential baseline, and C3 typed failure surfacing, scoped to the shared companion outcome contract the test suite already covers. The harness ships in [bench/](bench/): a runner that isolates each condition in its own Claude configuration directory and keeps verifiers outside the model visible worktree, a strict JSONL record schema, a summary script that refuses to compare snapshots from different task manifests, a redaction step for published transcripts, and a manifest tool that content hashes every task before any results exist. -No results are published yet. The task suite currently holds nine tasks (T01 to T09) of the planned 8 to 12. The first dated reference snapshot ships once the pool is complete, with raw per run data committed alongside the summary table. Until then the status is plain: the procedure is public and runnable, the numbers do not exist. Readers can run the harness themselves with `node bench/run.mjs --task --condition --repetition --results --claude-config ` after building condition profiles per [bench/conditions/README.md](bench/conditions/README.md). +No claim results are published yet. The task suite currently holds nine tasks (T01 to T09) of the planned 8 to 12. The first dated reference snapshot ships once the pool is complete, with raw per run data committed alongside the summary table. Until then the status is plain: the procedure is public and runnable, the numbers do not exist. Readers can run the harness themselves with `node bench/run.mjs --task --condition --repetition --results --claude-config ` after building condition profiles per [bench/conditions/README.md](bench/conditions/README.md). A separate lane instrument, `node bench/lanes/run.mjs`, runs one task brief through one engine companion on an explicit model and effort, so routing scores rest on matched evidence; its first snapshot is [bench/results/lanes-2026-10-13](bench/results/lanes-2026-10-13/summary.md), and it makes no claim under test. ## Data and uninstall The Codex plugin keeps job records, prompts, logs, event ledgers, and stored results under `~/.claude/plugins/data/codex-claude-code-fusion/`; prompts and logs can contain task text, diffs, command output, and diagnostics. Managed terminal history defaults to 256 records and 512 MiB, while running jobs, resume safety records, and uncollected background results are protected even when they temporarily exceed that quota. Unknown, corrupt, future-schema, and crash-orphaned content is left untouched rather than counted as safely managed history. Private raw-command staging is removed at SessionEnd for the matching session or after one hour during a later transport allocation. -The Grok plugin keeps its companion ledger, private job records, briefs, and logs under `~/.claude/plugins/data/grok-claude-code-fusion/`; briefs can contain prompts and diffs, logs can contain Grok stderr, and new records retain the CLI reported `usage` and `model_usage` objects. `/grok:history` reads a safe metadata projection from this ledger. The companion does not promise automatic garbage collection for terminal records, briefs, or logs, so clearing this history requires manually deleting that data directory after no running job, uncollected result, or resume evidence is needed. Native Grok conversations are separate under `~/.grok/sessions`; upstream retains them for 30 days by default. `~/.grok/config.toml` accepts a positive integer at `[storage] cleanup_ttl_days`; `0` falls back to the default 30 days. Cross-session memory is separate again under `~/.grok/memory` and can be read or written by upstream only when memory is enabled. Deleting the companion ledger does not delete native sessions or memory, and deleting native sessions does not clear companion history. A `model_usage` row contains input, output, cache-read, and model-call counts plus optional cost, so reporting never invents reasoning or total token channels. `usage_is_incomplete` means the usage ledger may have missed open subagents, usage application, or a drain timeout; present tokens are observed lower bounds and both job-total token and cost coverage are incomplete. Exact positive cost pairs use integer ticks at 10000000000 ticks per USD as authoritative. Grok raw-command staging uses one-time private directories under validated system temporary storage and is consumed before parsing or pruned after one hour. Its headless stdout uses a separate 0600 file that is unlinked as soon as it is opened. +The Grok plugin keeps its companion ledger, private job records, briefs, and logs under `~/.claude/plugins/data/grok-claude-code-fusion/`; briefs can contain prompts and diffs, logs can contain Grok stderr, and new records retain the CLI reported `usage` and `model_usage` objects. `/grok:history` reads a safe metadata projection from this ledger. The companion does not promise automatic garbage collection for terminal records, briefs, or logs, so clearing this history requires manually deleting that data directory after no running job, uncollected result, or resume evidence is needed. Native Grok conversations are separate under `~/.grok/sessions`; upstream retains them for 30 days by default. `~/.grok/config.toml` accepts a positive integer at `[storage] cleanup_ttl_days`; `0` falls back to the default 30 days. Cross-session memory is separate again under `~/.grok/memory` and can be read or written by upstream only when memory is enabled. Deleting the companion ledger does not delete native sessions or memory, and deleting native sessions does not clear companion history. A `model_usage` row contains input, output, cache-read, cache-creation, and model-call counts plus optional cost, so reporting never invents reasoning or total token channels. `usage_is_incomplete` means the usage ledger may have missed open subagents, usage application, or a drain timeout; present tokens are observed lower bounds and both job-total token and cost coverage are incomplete. Exact positive cost pairs use integer ticks at 10000000000 ticks per USD as authoritative. Grok raw-command staging uses one-time private directories under validated system temporary storage and is consumed before parsing or pruned after one hour. Its headless stdout uses a separate 0600 file that is unlinked as soon as it is opened. The Cursor plugin keeps its job records, briefs, logs, model catalog cache, compile cache, and each job's private home (including the Cursor chat store used for resume) under `~/.claude/plugins/data/cursor-claude-code-fusion/`; private homes contain chats that can hold task text and results. Cursor's own account side history follows the account's privacy mode. @@ -126,7 +126,7 @@ CI also runs `node bench/manifest.mjs --check`; contributors should run it after - `plugins/grok/`: the Grok integration (companion runtime, `grok-rescue` agent, `/grok:*` commands). - `plugins/cursor/`: the Cursor integration (companion runtime, `cursor-rescue` agent, `/cursor:*` commands). - `plugins/fusion/`: the orchestration layer: tier agents (`agents/`), the routing policy payload (`rules/`), and the `/fusion:panel`, `/fusion:setup`, `/fusion:smoke`, `/fusion:doctor`, `/fusion:stats`, `/fusion:ultra`, and `/fusion:config` commands. -- `bench/`: the benchmark methodology, harness, and task suite; no published results yet, see the publication gate in [bench/METHODOLOGY.md](bench/METHODOLOGY.md). +- `bench/`: the benchmark methodology, harness, task suite, and lane instrument; no claim results yet, see the publication gate in [bench/METHODOLOGY.md](bench/METHODOLOGY.md), and lane snapshots under `bench/results/lanes-/`. - `tests/`: the fake Codex, Grok, and Claude driven test suite. ## Internals diff --git a/SECURITY.md b/SECURITY.md index 2db7341..ad061ac 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -6,9 +6,9 @@ The Grok plugin executes the local Grok CLI with your xAI credentials. Consult, Before prompt delivery, the companion requires a new `ProfileApplied` event in the incremental delta of the shared `~/.grok/sessions/sandbox-events.jsonl` log (the home-root location that 1.0.13 and earlier wrote is unsupported) that contains this run's private temporary path and matches the canonical workspace, strict profile, enforced state, and requested `restrict_network: true` configuration. Upstream appends there with no rotation or size cap. The event proves that the profile and configuration were applied and marked enforced, not that the platform blocked every network path. Upstream records have no run id or pid, so the unique private path correlates positive evidence; unrelated `ProfileApplied` and `ApplyFailed` events are not attributed to the current run and failure events are auxiliary diagnostics only. A matching warning on the owned stderr stream, handshake timeout, missing or malformed matching evidence, shared-log rotation or disappearance, and matching field mismatches fail closed. Forced builder tracing initializes only after Grok consumes stdin, so it is not a pre-prompt or pre-side-effect attestation. Fallback or unmatched warnings trigger early termination, and a successful close is rejected without positive tool allowlist evidence. -The child sets all eighteen `GROK_CLAUDE_*_ENABLED`, `GROK_CURSOR_*_ENABLED`, and `GROK_CODEX_*_ENABLED` bridge variables to false; upstream currently consumes the six Claude and six Cursor cells plus the Codex sessions cell and reserves the other five Codex cells, and the child also pins `GROK_MANAGED_MCPS_ENABLED=false` because that variable has highest precedence over `[managed_mcps]` in `~/.grok/config.toml` and remote settings. It also pins `GROK_MANAGED_MCP_GATEWAY_TOOLS_ENABLED=false`, `GROK_AUTO_WAKE=false`, and `GROK_WEB_FETCH=1` with `--web` or `GROK_WEB_FETCH=0` without it. It disables subagents, disables automatic updates through `--no-auto-update`, removes inherited Claude session and plugin variables plus `_GROK_CLAUDE_MARKER_OVERRIDE`, `GROK_TEST_VERSION`, `GROK_SANDBOX`, `GROK_LEADER_SOCKET`, `GROK_SUBAGENTS_MAX_DEPTH`, `GROK_WEB_FETCH_PROXY`, and `GROK_WEB_FETCH_ALLOW_LOCAL`, and broadly scrubs secret-bearing variables while preserving required xAI authentication and organization version-policy variables. Grok enforces hard version requirements at startup regardless of `--no-auto-update` (introduced in 0.2.112, verified through 1.0.41; organization message, exit 1 before any envelope), which the companion classifies as `setup`. The companion's own tested interval is `[1.0.14, 1.0.31)`: below it managed runs fail closed as `setup`, at or past it `/grok:setup` reports the build as newer than the tested interval and a verification pass is advised; job records carry `grokVersion` so a breaker advisory can name the build that failed. Memory is force-disabled unless the user explicitly passes `--memory` to an ordinary task. Review and stop gate always keep memory off. A memory-enabled task allows upstream to inject selected global or workspace memory into the model context sent to xAI and to update memory according to the user's Grok configuration. Upstream automatic saving usually requires at least three real user prompts in the same resumed session, so one Fusion task usually only reads existing memory and does not guarantee that new memory is saved. Continuity defaults to manual. Optional `claude-session` affinity resumes only a compatible completed ordinary direct task from the same Claude session, exact resolved working directory, mode, strict profile, and memory boundary. Fusion routed briefs do not receive automatic affinity and stay fresh unless explicitly resumed; `--fresh` disables affinity for one task. Write includes terminal commands, so treat it as giving Grok your own shell inside the strict workspace view and run it on a clean branch or an orchestrator-created disposable worktree. Command-pattern denies for common direct grok, claude, and codex calls do not cover absolute paths, aliases or functions, or indirect scripts; a hard nested-execution boundary requires removing `run_terminal_cmd` or applying an OS-level executable or network policy. Headless stdout is captured through a 0600 file that is unlinked immediately after open. `grok --version` now prints `grok 1.0.30 ()`; the companion records the raw string and never parses it. Grok 1.0.0 introduces inline agent definitions (`--agent` and `--agents`), `--rules`, and `--system-prompt-override`, which the companion never passes, so managed runs are unaffected. A resumable Grok task keeps its strict profile, mode, and memory boundary for its lifetime, so incompatible resumes fail closed. Job briefs and logs are stored in plain text under `~/.claude/plugins/data/grok-claude-code-fusion/`. The routing policy, panel blindness, and circuit breaker are prompt level conventions, not runtime enforcement; see the Safety model section of the README for the full split between what code enforces and what prompts request. +The child sets all eighteen `GROK_CLAUDE_*_ENABLED`, `GROK_CURSOR_*_ENABLED`, and `GROK_CODEX_*_ENABLED` bridge variables to false; upstream currently consumes the six Claude and six Cursor cells plus the Codex sessions cell and reserves the other five Codex cells, and the child also pins `GROK_MANAGED_MCPS_ENABLED=false` because that variable has highest precedence over `[managed_mcps]` in `~/.grok/config.toml` and remote settings. It also pins `GROK_MANAGED_MCP_GATEWAY_TOOLS_ENABLED=false`, `GROK_AUTO_WAKE=false`, `GROK_TURN_SUMMARY=false` (the turn summary feature defaults on and makes an extra model call per turn), `GROK_FILE_ACCELERATION=false`, and `GROK_WEB_FETCH=1` with `--web` or `GROK_WEB_FETCH=0` without it. It disables subagents, disables automatic updates through `--no-auto-update`, removes inherited Claude session and plugin variables plus `_GROK_CLAUDE_MARKER_OVERRIDE`, `GROK_TEST_VERSION`, `GROK_SANDBOX`, `GROK_LEADER_SOCKET`, `GROK_SUBAGENTS_MAX_DEPTH`, `GROK_WEB_FETCH_PROXY`, `GROK_WEB_FETCH_ALLOW_LOCAL`, `GROK_SANDBOX_MODE`, `GROK_ARTIFACTS_BIND_REMOUNT`, `GROK_FILE_ACCELERATION_ROUTES`, `GROK_CURSOR_WORKER_API_BASE`, and `GROK_CURSOR_WORKER_COMMAND`, and broadly scrubs secret-bearing variables while preserving required xAI authentication and organization version-policy variables. Removing the web fetch proxy and local variables does not remove a proxy or local allowance configured in `~/.grok/config.toml`, which takes precedence over the environment. Grok enforces hard version requirements at startup regardless of `--no-auto-update` (introduced in 0.2.112, verified through 1.0.41; organization message, exit 1 before any envelope), which the companion classifies as `setup`. The companion's own tested interval is `[1.0.14, 1.0.42)`: below it managed runs fail closed as `setup`, at or past it `/grok:setup` reports the build as newer than the tested interval and a verification pass is advised; job records carry `grokVersion` so a breaker advisory can name the build that failed. Memory is force-disabled unless the user explicitly passes `--memory` to an ordinary task. Review and stop gate always keep memory off. A memory-enabled task allows upstream to inject selected global or workspace memory into the model context sent to xAI and to update memory according to the user's Grok configuration; since 1.0.52 a memory-enabled turn can also schedule memory maintenance after it ends, which can make further model calls. Upstream automatic saving usually requires at least three real user prompts in the same resumed session, so one Fusion task usually only reads existing memory and does not guarantee that new memory is saved. Continuity defaults to manual. Optional `claude-session` affinity resumes only a compatible completed ordinary direct task from the same Claude session, exact resolved working directory, mode, strict profile, and memory boundary. Fusion routed briefs do not receive automatic affinity and stay fresh unless explicitly resumed; `--fresh` disables affinity for one task. Write includes terminal commands, so treat it as giving Grok your own shell inside the strict workspace view and run it on a clean branch or an orchestrator-created disposable worktree. Command-pattern denies for common direct grok, claude, and codex calls do not cover absolute paths, aliases or functions, or indirect scripts; a hard nested-execution boundary requires removing `run_terminal_cmd` or applying an OS-level executable or network policy. Headless stdout is captured through a 0600 file that is unlinked immediately after open. `grok --version` now prints `grok 1.0.30 ()`; the companion records the raw string and never parses it. Grok 1.0.0 introduces inline agent definitions (`--agent` and `--agents`), `--rules`, and `--system-prompt-override`, which the companion never passes, so managed runs are unaffected. A resumable Grok task keeps its strict profile, mode, and memory boundary for its lifetime, so incompatible resumes fail closed. Job briefs and logs are stored in plain text under `~/.claude/plugins/data/grok-claude-code-fusion/`. The routing policy, panel blindness, and circuit breaker are prompt level conventions, not runtime enforcement; see the Safety model section of the README for the full split between what code enforces and what prompts request. -Upstream strict's `system_read` roots include `/var` and `/tmp` on Linux, and `/private` plus the entire `~/Library` on macOS. Shared temporary content and substantial user application configuration and caches may therefore be readable. Its write roots are the workspace and `~/.grok/sessions` (since 1.0.14 the rest of the Grok home is readable but not writable), always include `/tmp` and `/var/tmp`, and on macOS also include `/private/tmp`, `/private/var/tmp`, and all of `/private/var/folders`. The adapter's unique private `TMPDIR` is preferred and handshake-verified, not the only writable temporary surface. On Linux, a strict profile with deny paths re-executes under bubblewrap and refuses to start when `bwrap` is missing or unusable; macOS has no equivalent hard enforcement and relies on Seatbelt application plus best-effort tool denies. The sandbox permits consult `read_file` to reach `~/.grok/auth.json`, config, and sessions because strict keeps the Grok home readable, while write terminal commands can receive preserved xAI authentication variables. The companion applies best-effort Read denies to `auth.json`, `mcp_credentials.json`, `config.toml`, `sessions`, `memory`, `logs`, and `debug` under the Grok home with absolute paths and standard `**/.grok` patterns, but raw path variants, symbolic links, and shell or indirect scripts can bypass those tool-level rules. These Read denies apply to model tool calls. They do not prevent upstream's internal first-turn memory search and injection when `--memory` is enabled, so memory snippets can enter the xAI request even though direct model reads of `~/.grok/memory/**` are denied. Environment scrubbing and path-pattern denies do not isolate credentials or memory content. Model-facing MCP and meta-tool denies do not prove that native MCP servers, plugins, or hooks configured under `~/.grok` did not start during agent construction or had no side effects; those bridge variables disable compatibility imports, not native Grok configuration. Narrower exposure requires a dedicated sandbox profile, an isolated Grok home, an authentication broker, or upstream path-level authorization. +Upstream strict's `system_read` roots include `/var` and `/tmp` on Linux, and `/private` plus the entire `~/Library` on macOS. Shared temporary content and substantial user application configuration and caches may therefore be readable. Its write roots are the workspace and `~/.grok/sessions` (since 1.0.14 the rest of the Grok home is readable but not writable), always include `/tmp` and `/var/tmp`, and on macOS also include `/private/tmp`, `/private/var/tmp`, and all of `/private/var/folders`. The adapter's unique private `TMPDIR` is preferred and handshake-verified, not the only writable temporary surface. On Linux, a strict profile with deny paths re-executes under bubblewrap and refuses to start when `bwrap` is missing or unusable; macOS has no equivalent hard enforcement and relies on Seatbelt application plus best-effort tool denies. The sandbox permits consult `read_file` to reach `~/.grok/auth.json`, config, and sessions because strict keeps the Grok home readable, while write terminal commands can receive preserved xAI authentication variables. The companion applies best-effort Read denies to `auth.json`, `mcp_credentials.json`, `config.toml`, `sessions`, `memory`, `logs`, and `debug` under the Grok home with absolute paths and standard `**/.grok` patterns, but raw path variants, symbolic links, and shell or indirect scripts can bypass those tool-level rules. These Read denies apply to model tool calls. They do not prevent upstream's internal first-turn memory search and injection when `--memory` is enabled, so memory snippets can enter the xAI request even though direct model reads of `~/.grok/memory/**` are denied. Environment scrubbing and path-pattern denies do not isolate credentials or memory content. Model-facing MCP and meta-tool denies do not prove that native MCP servers, plugins, or hooks configured under `~/.grok` did not start during agent construction or had no side effects. The bridge variables gate only Grok's loose compatibility imports (skills, rules, agents, hooks, MCP servers, and sessions read from other tools' configuration), not native Grok configuration and not Claude Code plugins: unless `~/.grok/config.toml` sets `[claude_compat] imported = true`, Grok enables every plugin that `~/.claude/settings.json` enables, and since 1.0.52 an enabled and trusted plugin's hooks join the session at startup, so Claude Code plugin hooks and MCP servers can run inside a managed run before the first model call. Listing those plugins in `[plugins] disabled`, or setting `[claude_compat] imported = true`, keeps them out. Grok also passes the `env` block of `~/.claude/settings.json`, trusted project settings, and `.envrc` into tool subprocesses, so scrubbing the child environment does not keep those values out of write-mode shells. Startup can create directories and trust-boundary files inside the Grok home before confinement applies, so the sessions-only write grant describes the enforced phase. Narrower exposure requires a dedicated sandbox profile, an isolated Grok home, an authentication broker, or upstream path-level authorization. Under the runtime-socket deny policy (introduced between 1.0.4 and 1.0.13, still present at 1.0.41), strict refuses to start while `/var/run/docker.sock` is a symlink, which is the resting state of macOS Docker engines such as OrbStack; the companion preflights this and fails the dispatch as `sandbox` in seconds, and the repair is host side (for OrbStack `orb config set setup.use_admin false` and removing the link, for Docker Desktop disabling the default socket), never a sandbox downgrade. OrbStack's non-admin socket under `~/.orbstack/run` is outside upstream's deny list and the macOS strict profile allows network-outbound, so a strict write shell can drive that engine (verified at 1.0.41); the companion refuses write runs as `sandbox` while the socket exists, consult and review are unaffected, and the repair is quitting the engine or routing the package to Codex, whose sandbox blocks the connect. diff --git a/docs/grok-contract.md b/docs/grok-contract.md index 06bbbe6..95eaba8 100644 --- a/docs/grok-contract.md +++ b/docs/grok-contract.md @@ -16,7 +16,7 @@ Job statuses, liveness, rendered outcome footers, failure kind definitions, back - At 1.0.0, `grok --version` prints `grok 1.0.0 ()`. The companion records the raw string and never parses it. - The companion calls `grok --prompt-file /dev/stdin --output-format json` and pipes the complete prompt to the child only after sandbox startup is verified. Passing the companion's private brief path would fail under strict mode when Claude plugin data is outside the sandbox. Grok returns one JSON object on stdout. Its envelope identity keys `text`, `stopReason`, `sessionId`, `requestId`, `structuredOutput`, and `structuredOutputError` are camelCase, while top-level usage, cost, and flag fields are snake_case. Optional observed fields include `usage` (`input_tokens`, `cache_read_input_tokens`, `cache_creation_input_tokens`, `output_tokens`, `reasoning_tokens`, and `total_tokens`), `modelUsage`, `num_turns`, `total_cost_usd`, `total_cost_usd_ticks`, `cost_is_partial`, and `usage_is_incomplete`. The only incompleteness flags are `usage_is_incomplete` and `cost_is_partial`; there is no `cancellation-category` field or `model_usage_is_incomplete` field. Structured errors may also report a message and those observability fields, and they remain failures even if the CLI exits with code 0. The companion persists only reported fields and never invents missing values. Transient recoverable auth errors can appear on stderr, so the companion parses stdout and routes stderr to the per job log. Stdout is written to a 0600 file that is unlinked immediately after a successful open, parsed through the still open descriptor under a total byte limit, and closed after parsing; a bounded tail remains available for diagnostics without leaving a named spool on disk. A zero exit run without one valid JSON envelope maps to `transport`. -- An upstream `modelUsage` entry contains `input`, `output`, `cacheRead`, and `modelCalls`, plus optional `costUSD`. It does not contain reasoning or total token fields. Token stats preserve and aggregate only channels that the CLI actually reports and never synthesize aggregate reasoning or total tokens from per-model entries. Partial entries remain in the job record but are excluded from exact channel totals and counted as incomplete. Missing values are never filled with zero or derived from other fields, and coverage separates complete, incomplete, and unreported jobs. `usage_is_incomplete` means the upstream usage ledger may have missed open subagents, usage application, or a drain timeout. Reported token fields remain useful observed lower bounds, but the job-total token and cost coverage both fail closed to incomplete and those values never enter exact totals. Review is a single model call, so its usage and cost fields come directly from that call rather than merging a corrective resume. +- An upstream `modelUsage` entry contains `inputTokens`, `outputTokens`, `cacheReadInputTokens`, `cacheCreationInputTokens`, and `modelCalls`, plus optional `costUSD`; the companion keeps the input, output, cache read, and model call counts per model and the cache creation count in the aggregate usage. It does not contain reasoning or total token fields. Token stats preserve and aggregate only channels that the CLI actually reports and never synthesize aggregate reasoning or total tokens from per-model entries. Partial entries remain in the job record but are excluded from exact channel totals and counted as incomplete. Missing values are never filled with zero or derived from other fields, and coverage separates complete, incomplete, and unreported jobs. `usage_is_incomplete` means the upstream usage ledger may have missed open subagents, usage application, or a drain timeout. Reported token fields remain useful observed lower bounds, but the job-total token and cost coverage both fail closed to incomplete and those values never enter exact totals. Review is a single model call, so its usage and cost fields come directly from that call rather than merging a corrective resume. - Stats expose headless turn and cost coverage separately from token coverage. Reported turn counts are summed without treating missing counts as zero. A cost enters the exact total only when the top-level USD and tick values are both finite and positive, `cost_is_partial` is not true, `usage_is_incomplete` is not true, and the companion-derived `modelUsageIsIncomplete` flag is not true; partial and unreported jobs stay in separate buckets. Tick values are authoritative at 10000000000 ticks per USD, so display and aggregation use ticks rather than accumulating floating point `costUSD` rows. - Model and effort are never passed by default; grok resolves them from its own config (`~/.grok/config.toml`, project `.grok/config.toml`). The companion's `--model` and `--effort` flags forward them explicitly when a caller insists. Model capture uses envelope model fields first, then a sole `modelUsage` key, then the session summary model, including salvaged error envelopes, while effort remains observable only when the CLI reports it. Requested and resolved values remain distinct. - Wrapper Agents never interpolate the raw user brief into Bash. They allocate a private token-named staging file bound to the creating Claude session, write the opaque request through the Write tool, and invoke a fixed companion argv containing only the validated token. The companion validates ownership, permissions, age, session binding, file identity, size, and UTF-8 before consuming and deleting that file, then parses options. SessionEnd immediately removes verified unused transports owned by the ending Claude session without applying the age gate. Every later transport allocation globally removes verified private transports owned by the current UID once they are older than one hour, regardless of their recorded session, so a crashed session cannot retain request data indefinitely. Symlinks, widened permissions, different ownership, malformed metadata, and fresh transports are never removed by that sweep. The companion persists the parsed brief in private plugin state for lifecycle and resume evidence. Internal task, review, and stop-gate calls all deliver the child prompt through stdin. @@ -44,11 +44,11 @@ Job statuses, liveness, rendered outcome footers, failure kind definitions, back - Consult mode pins `--sandbox strict --permission-mode default` and passes `--tools read_file,grep,list_dir`. `--web` extends that hard built in tool filter with `web_search,web_fetch`; without it, `--disable-web-search` is required. `x_search` is distinct from `web_search` and `web_fetch`, and `--disable-web-search` does not remove it, but the fixed `--tools` allowlist keeps it out of managed runs, as verified by a live 1.0.0 no-web consult that reported exactly `read_file`, `list_dir`, and `grep`. The companion adds no speculative disallow entries because Grok 1.0.0 warns when a `disallowedTools` entry matches nothing and the companion's tracing fails closed on that warning. Write mode also pins `--sandbox strict` and auto approves a fixed allowlist containing `read_file`, `grep`, `list_dir`, `search_replace`, and `run_terminal_cmd`, plus direct web tools only when requested. Upstream 1.0.0 ships an OpenCode-compatible `write` tool id in its default toolsets, but the companion's fixed `--tools` allowlist keeps it, and every other new default tool including `monitor`, `workflow`, `todo_write`, `scheduler`, media tools, and `task` or `spawn_subagent`, out of managed runs. `search_replace` remains the file-creation path in the write set. Explicit denies reject Edit, Write, and Bash in consult and reject `search_tool`, `use_tool`, `ask_user_question`, and all MCP tools in every mode. Every call also passes bare `--disallowed-tools Agent`, sets `GROK_SUBAGENTS=0`, and retains `--no-subagents` only as a compatibility hint. `GROK_MEMORY=0` applies unless an ordinary task explicitly enables memory, when that task receives `GROK_MEMORY=1`. Upstream parses `--no-subagents`, but the single-turn and agent resolvers do not forward it, while the interactive TUI does apply it; the hard Agent tool deny and `GROK_SUBAGENTS=0` are the effective headless controls. The single-turn resolver also does not forward `--no-ask-user`, `--no-plan`, `--experimental-memory`, `--no-memory`, or `--worktree-ref`. - The open-source snapshot is per-crate versioned, unversioned as a release train with no tags, and can lag the installed binary, whose stable channel ships weekly. The companion probes the installed binary instead of trusting semantic version or the published source snapshot. Every managed run requires `--prompt-file`, `--output-format`, `--sandbox`, `--tools`, `--disallowed-tools`, `--deny`, `--max-turns`, `--no-auto-update`, and `--no-wait-for-background`. Consult additionally requires `--permission-mode` and `--allow`; no-web runs require `--disable-web-search`; write requires `--always-approve`; review requires `--json-schema`. Required flags must appear in the effective capability surface before launch. Required flags hidden from `--help` (currently `--no-auto-update` and `--no-wait-for-background`, `clap hide=true` at least from 0.2.110 through 1.0.0) are verified by a per-flag acceptance probe (`<flag> --help`, or `<flag>=<dummy> --help` for value flags, with exit 0) instead of help-text presence, and this probe covers every required flag. The companion never silently drops a requested mode or downgrades strict to workspace. - Every run forces `RUST_LOG=xai_grok_agent::builder=debug,xai_grok_sandbox=warn,xai_grok_shell::session::acp_session::turn=warn`. The `xai_grok_sandbox=warn` directive carries upstream sandbox failure warnings to the owned stderr stream, and the turn directive carries upstream's `auth 401 retry: no credential was sent` warning; on its first occurrence the companion terminates the run and fails it as `auth` instead of letting grok's re-authentication loop (up to 50 resubmits with backoff) run into the timeout. Under strict that loop means the stored OIDC token expired and could not be refreshed, because grok 1.0.14 and later can write only `~/.grok/sessions` inside the Grok home; the companion therefore refreshes a token older than five hours with one `grok models` call outside the sandbox before a managed run, and after an `auth` failure refreshes once and reruns once; an `auth` failure after that means the refresh token itself expired, and the repair is `grok login`, never a sandbox downgrade. Upstream initializes builder tracing only after it has consumed the complete stdin prompt, so tool policy cannot be attested before prompt delivery or before every possible side effect. A fallback or unmatched warning triggers immediate verified termination as soon as it appears, and a nominally successful close is rejected unless positive `tools allowlist applied` evidence was observed. An unmappable tool id makes Grok retain the full tool surface, and an unmatched disallowed rule may hide a misspelled boundary. Consult retains independent explicit Bash, Edit, Write, MCP, Agent, and meta-tool denies; write is already a user-authorized modification surface. -- The child sets all eighteen `GROK_CLAUDE_*_ENABLED`, `GROK_CURSOR_*_ENABLED`, and `GROK_CODEX_*_ENABLED` bridge variables to false; upstream currently consumes the six Claude and six Cursor cells plus the Codex sessions cell and reserves the other five Codex cells, and the child also pins `GROK_MANAGED_MCPS_ENABLED=false` because that variable has highest precedence over `[managed_mcps]` in `~/.grok/config.toml` and remote settings. It also pins `GROK_MANAGED_MCP_GATEWAY_TOOLS_ENABLED=false` and `GROK_AUTO_WAKE=false` (the upstream default is true), removes inherited `CLAUDE_CODE_*` and `CLAUDE_PLUGIN_*` variables, deletes `_GROK_CLAUDE_MARKER_OVERRIDE`, `GROK_TEST_VERSION`, `GROK_SANDBOX`, `GROK_LEADER_SOCKET`, and `GROK_SUBAGENTS_MAX_DEPTH`, and broadly scrubs secret-bearing environment variables except the xAI authentication material required by Grok. Organization version-policy variables including `GROK_REQUIRED_MINIMUM_VERSION` are deliberately preserved. Automatic updates are disabled with `--no-auto-update`. Grok splits soft update floors and ceilings from hard startup requirements (introduced in 0.2.112, verified through 1.0.30 on 2026-09-14): `[cli] minimum_version` and `maximum_version`, plus `GROK_MINIMUM_VERSION` and `GROK_MAXIMUM_VERSION`, steer auto-update only, while `[cli] required_minimum_version` and `required_maximum_version`, plus `GROK_REQUIRED_MINIMUM_VERSION` and `GROK_REQUIRED_MAXIMUM_VERSION`, can resolve from remote settings. The hard layer exits 1 with an organization message on stderr before the headless turn, `--no-auto-update` does not bypass it, and the companion classifies that death as `setup`. +- The child sets all eighteen `GROK_CLAUDE_*_ENABLED`, `GROK_CURSOR_*_ENABLED`, and `GROK_CODEX_*_ENABLED` bridge variables to false; upstream currently consumes the six Claude and six Cursor cells plus the Codex sessions cell and reserves the other five Codex cells, and the child also pins `GROK_MANAGED_MCPS_ENABLED=false` because that variable has highest precedence over `[managed_mcps]` in `~/.grok/config.toml` and remote settings. It also pins `GROK_MANAGED_MCP_GATEWAY_TOOLS_ENABLED=false` and `GROK_AUTO_WAKE=false` (the upstream default is true), `GROK_TURN_SUMMARY=false` (the turn summary feature defaults on and adds a grok-4.5 model call per turn), and `GROK_FILE_ACCELERATION=false`, removes inherited `CLAUDE_CODE_*` and `CLAUDE_PLUGIN_*` variables, deletes `_GROK_CLAUDE_MARKER_OVERRIDE`, `GROK_TEST_VERSION`, `GROK_SANDBOX`, `GROK_LEADER_SOCKET`, `GROK_SUBAGENTS_MAX_DEPTH`, and the 1.0.52 surfaces `GROK_SANDBOX_MODE` (an extra per command sandbox), `GROK_ARTIFACTS_BIND_REMOUNT` (a bind time command), `GROK_FILE_ACCELERATION_ROUTES`, `GROK_CURSOR_WORKER_API_BASE`, and `GROK_CURSOR_WORKER_COMMAND`, and broadly scrubs secret-bearing environment variables except the xAI authentication material required by Grok. Organization version-policy variables including `GROK_REQUIRED_MINIMUM_VERSION` are deliberately preserved. Automatic updates are disabled with `--no-auto-update`. Grok splits soft update floors and ceilings from hard startup requirements (introduced in 0.2.112, verified through 1.0.30 on 2026-09-14): `[cli] minimum_version` and `maximum_version`, plus `GROK_MINIMUM_VERSION` and `GROK_MAXIMUM_VERSION`, steer auto-update only, while `[cli] required_minimum_version` and `required_maximum_version`, plus `GROK_REQUIRED_MINIMUM_VERSION` and `GROK_REQUIRED_MAXIMUM_VERSION`, can resolve from remote settings. The hard layer exits 1 with an organization message on stderr before the headless turn, `--no-auto-update` does not bypass it, and the companion classifies that death as `setup`. -- The companion's own tested interval is `[1.0.14, 1.0.31)`: below it managed runs fail closed as `setup`, at or past it `/grok:setup` reports the build as newer than the tested interval and a verification pass is advised; job records carry `grokVersion` so a breaker advisory can name the build that failed. +- The companion's own tested interval is `[1.0.14, 1.0.42)`: below it managed runs fail closed as `setup`, at or past it `/grok:setup` reports the build as newer than the tested interval and a verification pass is advised; job records carry `grokVersion` so a breaker advisory can name the build that failed. - Strict's upstream `system_read` surface is broader than the workspace and Grok home. Linux includes `/var` and `/tmp`; macOS includes `/private` and additionally exposes the entire `~/Library`, so shared temporary content and substantial user application configuration and caches may be readable. Its write roots are the workspace and `~/.grok/sessions` (since 1.0.14 the rest of the Grok home is readable but not writable), always include `/tmp` and `/var/tmp`, and on macOS also include `/private/tmp`, `/private/var/tmp`, and all of `/private/var/folders`; the companion adds its unique run `TMPDIR` and verifies that it appears in the applied profile. The private directory is the adapter's preferred temporary path, not the only writable temporary surface. On Linux, a strict profile with deny paths re-executes under bubblewrap and refuses to start when `bwrap` is missing or unusable; macOS has no equivalent hard enforcement and relies on Seatbelt application plus best-effort tool denies. User toolchains outside strict's readable paths may be unavailable in write runs. Strict requests child process network restriction. Linux enforces it through seccomp, while the current macOS network blocker is a no-op, so write shell commands can still reach the network there. `--web` controls only Grok's built in web tools and may transmit inspected content to xAI services; it is not a shell network gate. The companion pins `GROK_WEB_FETCH=1` with `--web` and `GROK_WEB_FETCH=0` without it, and scrubs `GROK_WEB_FETCH_PROXY` and `GROK_WEB_FETCH_ALLOW_LOCAL`. The companion never retries under workspace or treats a missing host tool as permission to weaken the profile. Grok's memory-tracing diagnostics and models-cache refresh resolve the real OS home independently of a `$HOME` override on macOS (introduced in 0.2.112, verified through 1.0.30 on 2026-09-14; observed 2026-07-29), so redirecting `HOME` does not fully relocate `~/.grok` side effects; companion behavior is unaffected because it never overrides `HOME`. -- The Grok home is an upstream trust boundary, not private adapter state. The sandbox permits consult's `read_file` tool to reach files such as `~/.grok/auth.json`, configuration, and sessions, while write's terminal can receive the retained xAI authentication environment. Since 1.0.14 strict writes only `~/.grok/sessions` inside the home, so those files are readable but not writable from a managed run. The companion layers best-effort Read denies over `auth.json`, `mcp_credentials.json`, `config.toml`, `sessions`, `memory`, `logs`, and `debug` under the Grok home, using both absolute paths and standard `**/.grok` patterns. Raw path variants, symbolic links, and shell or indirect scripts can bypass those tool-level patterns, so they are exposure reduction rather than credential isolation. The Read denies apply to model tools and do not block upstream's internal first-turn memory injection when `--memory` is enabled. Scrubbing unrelated environment secrets does not isolate filesystem credentials, opted-in memory content, or the authentication Grok itself needs. Grok's builder specially preserves MCP meta-tools; the explicit `search_tool`, `use_tool`, and MCP denies block model invocation but cannot prove that native MCP servers, plugins, or hooks configured under `~/.grok` did not start during agent construction or had no side effects. Those bridge variables disable compatibility bridges, not native Grok configuration. Narrower exposure requires a dedicated sandbox profile, an isolated Grok home, an authentication broker, or upstream path-level authorization; the current companion does not claim a hard path boundary. Grok offers configuration-layer narrowing primitives the companion does not currently inject per run (introduced in 0.2.112, verified through 1.0.30 on 2026-09-14): `[shell_environment_policy]` for the environment reaching shell tools, `[toolset.web_fetch] allowed_domains`, and hosted-tool `tool_overrides` domain and date fences. +- The Grok home is an upstream trust boundary, not private adapter state. The sandbox permits consult's `read_file` tool to reach files such as `~/.grok/auth.json`, configuration, and sessions, while write's terminal can receive the retained xAI authentication environment. Since 1.0.14 strict writes only `~/.grok/sessions` inside the home, so those files are readable but not writable from a managed run. The companion layers best-effort Read denies over `auth.json`, `mcp_credentials.json`, `config.toml`, `sessions`, `memory`, `logs`, and `debug` under the Grok home, using both absolute paths and standard `**/.grok` patterns. Raw path variants, symbolic links, and shell or indirect scripts can bypass those tool-level patterns, so they are exposure reduction rather than credential isolation. The Read denies apply to model tools and do not block upstream's internal first-turn memory injection when `--memory` is enabled. Scrubbing unrelated environment secrets does not isolate filesystem credentials, opted-in memory content, or the authentication Grok itself needs. Grok's builder specially preserves MCP meta-tools; the explicit `search_tool`, `use_tool`, and MCP denies block model invocation but cannot prove that native MCP servers, plugins, or hooks configured under `~/.grok` did not start during agent construction or had no side effects. The bridge variables gate only the loose compatibility imports, not native Grok configuration and not Claude Code plugins: unless `~/.grok/config.toml` sets `[claude_compat] imported = true`, Grok enables every plugin that `~/.claude/settings.json` enables, and since 1.0.52 an enabled and trusted plugin's hooks join the session at startup alongside its MCP servers, so they can run inside a managed run before the first model call. `[plugins] disabled` or `[claude_compat] imported = true` in `~/.grok/config.toml` keeps them out; the companion does not rewrite that file. Grok also passes the `env` block of `~/.claude/settings.json`, trusted project settings, and `.envrc` into tool subprocesses, so child environment scrubbing does not keep those values out of write-mode shells, and removing the web fetch proxy and local variables does not remove an allowance configured in TOML, which takes precedence over the environment. Startup can create directories and trust-boundary files inside the Grok home before confinement applies, so the sessions-only write grant describes the enforced phase. Narrower exposure requires a dedicated sandbox profile, an isolated Grok home, an authentication broker, or upstream path-level authorization; the current companion does not claim a hard path boundary. Grok offers configuration-layer narrowing primitives the companion does not currently inject per run (introduced in 0.2.112, verified through 1.0.30 on 2026-09-14): `[shell_environment_policy]` for the environment reaching shell tools, `[toolset.web_fetch] allowed_domains`, and hosted-tool `tool_overrides` domain and date fences. - Before every managed launch, the companion creates a unique private `TMPDIR`, snapshots Grok's shared `~/.grok/sessions/sandbox-events.jsonl` log (the home-root location that 1.0.13 and earlier wrote is unsupported), starts the child, and withholds the prompt until a new `ProfileApplied` event containing that private path also proves the expected canonical workspace, `strict` profile, enforced state, and requested `restrict_network: true` configuration. At 1.0.0, `ProfileApplied` also carries `platform` (`linux` with `landlock`, or `macos` with `seatbelt`), and the log can add `FsViolation`, `NetViolation`, `BypassGranted`, and `BypassDenied` event types. Those fields and events are additive, so the companion matcher is unaffected. There is no dedicated tmpdir field: temporary roots ride inside `read_write_paths`, which is exactly what the companion matches. Upstream events have no run id or pid, so the unique `TMPDIR` is the correlation key for positive evidence; unrelated `ProfileApplied` and `ApplyFailed` records are not attributed to the current run, and failure records remain auxiliary diagnostics. The matching event proves that the profile and configuration were applied and marked enforced, not that the platform blocked every network path. Upstream appends to `~/.grok/sessions/sandbox-events.jsonl` with no rotation or size cap, so the file grows unbounded; the companion's rotation and truncation detection therefore signals external interference, not normal upstream behavior. A matching warning on the owned stderr stream, a handshake timeout, missing or malformed matching evidence, shared-log disappearance or rotation, or any matching field mismatch terminates the owned process and fails closed as `sandbox`. Stderr can fail the owned handshake but cannot prove successful enforcement. - Upstream wires only `bypassPermissions` at spawn; the remaining `--permission-mode` values are forward-compatibility placeholders. The companion still passes `default` in headless consult runs, where a call that would prompt is cancelled, but consult isolation rests on the hard `--tools` filter and deny rules rather than permission mode. - `GROK_CONSULT_ALLOW` is retained only for read, grep, web search, and web fetch permission patterns. Shell, edit, MCP, and Agent rules are discarded, and the variable cannot add a tool omitted by the hard `--tools` filter. @@ -81,7 +81,7 @@ Job statuses, liveness, rendered outcome footers, failure kind definitions, back - `GROK_COMPANION_RUNTIME_SOCKET_CANDIDATES`: colon separated runtime-socket deny paths for managed-run preflight. When unset, the companion checks its default Docker, Podman, and containerd endpoints. An explicit empty value disables candidate checks, which test fixtures use to remain independent of host Docker state. -- `GROK_BIN`: Grok binary override (tests point it at a fake). `GROK_COMPANION_DATA`: absolute state directory override (default `~/.claude/plugins/data/grok-claude-code-fusion`); relative values are rejected. `GROK_COMPANION_CONTINUITY_POLICY`: `manual` or `claude-session`, overriding the persisted setup choice. `GROK_COMPANION_TIMEOUT_MS`: foreground timeout override. `GROK_COMPANION_BACKGROUND_DELIVERY`: internal value `managed` marks a detached review that its dedicated runner owns and collects; unset explicit background jobs are manual. `GROK_JOBS_MONITOR_MANAGED_GRACE_MS`: delayed fallback window for an uncollected managed result, default 60000ms. `GROK_COMPANION_PIDLESS_RUNNING_GRACE_MS`: pidless launcher grace override. `GROK_COMPANION_WAIT_POLL_MS`: result wait poll interval override. `GROK_COMPANION_WAIT_TIMEOUT_MS`: result wait budget override, clamped to 570000ms. `GROK_COMPANION_SANDBOX_HANDSHAKE_MS`: sandbox event handshake budget, capped by the runtime maximum. `GROK_CONSULT_ALLOW`: comma separated read, grep, web search, or web fetch permission patterns; it cannot expand the hard consult tool set and write mode ignores it. The child receives a unique private `TMPDIR`, an explicit `GROK_MEMORY=0` or `1` selected by the ordinary task contract, `GROK_SUBAGENTS=0`, all eighteen disabled `GROK_CLAUDE_*_ENABLED`, `GROK_CURSOR_*_ENABLED`, and `GROK_CODEX_*_ENABLED` bridge variables, `GROK_MANAGED_MCPS_ENABLED=false`, `GROK_MANAGED_MCP_GATEWAY_TOOLS_ENABLED=false`, `GROK_AUTO_WAKE=false`, and forced builder tracing. With `--web`, it receives `GROK_WEB_FETCH=1`; without it, `GROK_WEB_FETCH=0`. Inherited Claude session and plugin variables, `_GROK_CLAUDE_MARKER_OVERRIDE`, `GROK_TEST_VERSION`, `GROK_SANDBOX`, `GROK_LEADER_SOCKET`, `GROK_SUBAGENTS_MAX_DEPTH`, `GROK_WEB_FETCH_PROXY`, `GROK_WEB_FETCH_ALLOW_LOCAL`, inherited `GROK_MEMORY`, and secret-bearing variables unrelated to xAI authentication cannot override those controls. Organization version-policy variables remain preserved. +- `GROK_BIN`: Grok binary override (tests point it at a fake). `GROK_COMPANION_DATA`: absolute state directory override (default `~/.claude/plugins/data/grok-claude-code-fusion`); relative values are rejected. `GROK_COMPANION_CONTINUITY_POLICY`: `manual` or `claude-session`, overriding the persisted setup choice. `GROK_COMPANION_TIMEOUT_MS`: foreground timeout override. `GROK_COMPANION_BACKGROUND_DELIVERY`: internal value `managed` marks a detached review that its dedicated runner owns and collects; unset explicit background jobs are manual. `GROK_JOBS_MONITOR_MANAGED_GRACE_MS`: delayed fallback window for an uncollected managed result, default 60000ms. `GROK_COMPANION_PIDLESS_RUNNING_GRACE_MS`: pidless launcher grace override. `GROK_COMPANION_WAIT_POLL_MS`: result wait poll interval override. `GROK_COMPANION_WAIT_TIMEOUT_MS`: result wait budget override, clamped to 570000ms. `GROK_COMPANION_SANDBOX_HANDSHAKE_MS`: sandbox event handshake budget, capped by the runtime maximum. `GROK_CONSULT_ALLOW`: comma separated read, grep, web search, or web fetch permission patterns; it cannot expand the hard consult tool set and write mode ignores it. The child receives a unique private `TMPDIR`, an explicit `GROK_MEMORY=0` or `1` selected by the ordinary task contract, `GROK_SUBAGENTS=0`, all eighteen disabled `GROK_CLAUDE_*_ENABLED`, `GROK_CURSOR_*_ENABLED`, and `GROK_CODEX_*_ENABLED` bridge variables, `GROK_MANAGED_MCPS_ENABLED=false`, `GROK_MANAGED_MCP_GATEWAY_TOOLS_ENABLED=false`, `GROK_AUTO_WAKE=false`, `GROK_TURN_SUMMARY=false`, `GROK_FILE_ACCELERATION=false`, and forced builder tracing. With `--web`, it receives `GROK_WEB_FETCH=1`; without it, `GROK_WEB_FETCH=0`. Inherited Claude session and plugin variables, `_GROK_CLAUDE_MARKER_OVERRIDE`, `GROK_TEST_VERSION`, `GROK_SANDBOX`, `GROK_LEADER_SOCKET`, `GROK_SUBAGENTS_MAX_DEPTH`, `GROK_WEB_FETCH_PROXY`, `GROK_WEB_FETCH_ALLOW_LOCAL`, `GROK_SANDBOX_MODE`, `GROK_ARTIFACTS_BIND_REMOUNT`, `GROK_FILE_ACCELERATION_ROUTES`, `GROK_CURSOR_WORKER_API_BASE`, `GROK_CURSOR_WORKER_COMMAND`, inherited `GROK_MEMORY`, and secret-bearing variables unrelated to xAI authentication cannot override those controls. Organization version-policy variables remain preserved. ## Authentication refresh diff --git a/plugins/fusion/rules-manifest.json b/plugins/fusion/rules-manifest.json index ce872c4..d4731bb 100644 --- a/plugins/fusion/rules-manifest.json +++ b/plugins/fusion/rules-manifest.json @@ -33,6 +33,7 @@ "9e3213d69a08713ffa97f22042ecee8493fb4aa4a2d91150da4f1b6e4573b42a", "a04a6b16fdf5ff42e8ab8da398331da505bdffa0a002e3073c4e73d3298bd3fc", "a53ecbb4a64794840a7efadd34c383fec8851ffde8e3b2380f027baa023b1f46", + "a7ea340df78f0de3dc72d5395fa2ea10013232aa48fdccbd56e7ebc92a6f8f0b", "a90f49b6ce745c9184bd11509418ca798b26f88351fbb5d9322975fc3c2a94e1", "adf8e377d4ab99a4c1dcc59f8f8d608def9b1f347b061c73318834b285c09c34", "aead643e6dee6b0de04e97c5dbbb2f9f8c5e04dfbd12e8305b0abb97ab33524e", diff --git a/plugins/fusion/rules/orchestration.md b/plugins/fusion/rules/orchestration.md index 03c0489..fda315d 100644 --- a/plugins/fusion/rules/orchestration.md +++ b/plugins/fusion/rules/orchestration.md @@ -72,11 +72,11 @@ A capability monopoly hard routes: when exactly one lane possesses the load bear | gpt-6-luna | codex | openai | 3 | 3 | 5 | GPT-6 volume tier, shipped 2026-09-22 and the codex volume lane default; Codex credits 2.5 input, 0.25 cached and 12.5 output per M tokens, half of gpt-5.6-luna; 146 tok/s and 18s to first token at xhigh; Coding Agent Index 41 at max vs 43 for gpt-5.6-luna, so it trades a little quality for half the cost; priority tier listed as 1.5x speed with no increased usage note; no ultra effort | | grok-4.7 | grok | xai | 4 | 3 | 4 | Grok flagship weights, shipped 2026-09-21 on a new larger base; AA Coding Agent Index 56 at xhigh in Grok Build vs 47 for grok-4.6, Terminal-Bench 4.0 about 38 vs 20, AA Intelligence v4.3.2 46; same list price as grok-4.6 ($2 input, $0.50 cached, $6 output per M under 200k) but about 81k output tokens per index task vs 36k; on its standard serving path high effort research consults took 339 to 495s and two scoped reviews timed out at the 570s flight on 2026-09-27, so the foreground consult roles default to grok-4.7-build-fast, the same weights on faster serving; X and live platform reads remain a lane monopoly (verified 2026-07-26); taste unvalidated | | grok-4.7-build-fast | grok | xai | 4 | 3 | 3 | Same weights as grok-4.7 served at about twice the output speed for twice the token price, Grok Build and Cursor only; the grok burst (low), independence (high) and live-web (high) lane default because foreground runs must fit the 570s flight: 60 to 115s typical at low and 167s for a live-web consult at high on 2026-09-27; open ended verification briefs still run long (one scoped security review timed out at high), so review briefs stay narrow | -| composer-2.5 | cursor | cursor | 3 | 3 | 5 | Cursor's own model (Moonshot Kimi K2.5 base with Cursor RL, released 2026-05-18), served only through Cursor from the first-party pool at $0.50 input and $2.50 output per M (composer-2.5-fast $3 and $15); the cursor burst lane default; SWE-Bench Multilingual 79.8 and AA Coding Agent Index 63 in the Cursor CLI at launch on an older index version; a live consult took 20s end to end on 2026-10-13; provisional scores, promotion past grok burst rests on matched acceptance evidence; taste unvalidated | -| muse-spark-1.3 | cursor | meta | 4 | 3 | 3 | Meta flagship through Cursor's API pool, catalog variants minimal to max with a 1M window; the cursor diversity lane default at xhigh because Meta weights sit outside every other lane; AA Coding Agent Index 54.3 at max and 48.3 at xhigh in the September 2026 snapshot; provisional scores; taste unvalidated | -| gemini-3.8-flash | cursor | google | 3 | 3 | 4 | Google flash through Cursor's API pool with a 1M window and low, medium, and high variants; the second unseated family for diversity when Meta already sits on a question; unbenchmarked here; provisional scores; taste unvalidated | -| kimi-k3 | cursor | moonshot | 3 | 3 | 4 | Moonshot flagship through Cursor's API pool with a 1M window and low, high, and max variants; AA Coding Agent Index 51.9 in Kimi Code CLI in the September 2026 snapshot; shares lineage with composer-2.5, so never seat both on one question; provisional scores; taste unvalidated | -| glm-5.2 | cursor | zhipu | 3 | 3 | 4 | Zhipu GLM through Cursor's API pool with high and max variants; unbenchmarked here; provisional scores; taste unvalidated | +| composer-2.5 | cursor | cursor | 3 | 3 | 5 | Cursor's own model (Moonshot Kimi K2.5 base with Cursor RL, released 2026-05-18), served only through Cursor from the first-party pool at $0.50 input and $2.50 output per M; the cursor burst lane default; 2026-10-13 lane snapshot: T05 and T09 6 of 6 at a 68 and 75s median, T06 0 of 3 because its CSV fix stops at what the brief spells out, where gpt-6-luna at xhigh passed 9 of 9 at 105 to 131s; no promotion past grok burst (no grok measurement) or toward the codex volume tiers; composer-2.5-fast ($3 and $15) reached the same verdicts, so it holds no seat; taste 3 rests on T09 style fidelity 3 of 3 | +| muse-spark-1.3 | cursor | meta | 4 | 3 | 3 | Meta flagship through Cursor's API pool, catalog variants minimal to max with a 1M window; the cursor diversity lane default at xhigh because Meta weights sit outside every other lane; AA Coding Agent Index 54.3 at max and 48.3 at xhigh in the September 2026 snapshot; 2026-10-13 lane snapshot: matched deep-reasoner on four blind consults, a latent defect in the T01 reference solution included, with no false claims at 32 to 68s against 65 to 221s; taste unvalidated | +| gemini-3.8-flash | cursor | google | 3 | 3 | 4 | Google flash through Cursor's API pool with a 1M window and low, medium, and high variants; 2026-10-13 lane snapshot at high: every key finding plus one false defect on the clean review, and the slowest seat at 71 to 230s, so it ranks below muse-spark-1.3 and kimi-k3 for diversity; taste unvalidated | +| kimi-k3 | cursor | moonshot | 4 | 3 | 4 | Moonshot flagship through Cursor's API pool with a 1M window and low, high, and max variants; AA Coding Agent Index 51.9 in Kimi Code CLI in the September 2026 snapshot; 2026-10-13 lane snapshot at high: matched deep-reasoner on four blind consults, the latent T01 defect included, with no false claims and the fastest seat at 27 to 35s; shares lineage with composer-2.5, so never seat both on one question; taste unvalidated | +| glm-5.2 | cursor | zhipu | 3 | 3 | 4 | Zhipu GLM through Cursor's API pool with high and max variants; 2026-10-13 lane snapshot at high: missed the latent T01 defect while asserting none existed and cited two lines one off, otherwise matched the key at 26 to 44s, so it ranks below muse-spark-1.3 and kimi-k3 for diversity; taste unvalidated | Lane defaults: codex quick gpt-6-sol@xhigh, volume gpt-6-luna@xhigh, flagship gpt-6-astra@xhigh; grok burst grok-4.7-build-fast@low, independence grok-4.7-build-fast@high, live-web grok-4.7-build-fast@high, large-context grok-4.6@medium; cursor diversity muse-spark-1.3@xhigh, burst composer-2.5@default. @@ -187,9 +187,9 @@ Every brief is self contained: goal, constraints, relevant paths, and what done - Related follow up packages reuse the existing worker thread via SendMessage instead of cold starting a new agent, unless the thread rotation rules in the fusion plugin's troubleshooting rules (shipped with the plugin, consulted on demand) say otherwise. A follow up sent to a peer wrapper restates the leading companion options (`--model`, `--effort`, `--cwd`) exactly as a fresh dispatch would, because the continuation composes a new engine call and otherwise falls silently to the CLI config default. A continuation of a finished peer wrapper or Claude worker reopens its Fusion task, and the round is collected and settled like a fresh dispatch: for a peer wrapper the lifecycle fills an omitted `--model` or `--effort` from the lane defaults and the new engine job joins the task's job history, and a Claude worker round runs on a fresh budget measured from the reopen. - Parallel packages declare each other: every brief names sibling packages' files as intended in flight changes and forbids reverting, restoring, or cleaning anything outside its own list. A worker's end state check covers its own files only. - A worktree isolated peer package is collected by merging its diff back into the main tree and re-running its verification command there before the package counts as done; for Codex, record semantic acceptance only after that main tree verification. The orchestrator creates a real Git worktree first and launches the peer with that canonical directory as its actual process cwd. Grok headless `--worktree` and `--worktree-ref` are not substitutes: headless `-p --worktree` creates a real worktree since 1.0.19, but the companion keys its handshake, state directory, and collection to the cwd it launched, so an engine-created worktree is invisible to Fusion. Worktree runs leave disposable per-path state directories in the engine's data root, and /fusion:stats --prune-dead is the cleanup path once the worktree itself is gone. After each release, `/fusion:stats --prune-dead` runs as part of the release cadence. Worktrees hosting detached peer jobs are created and removed by the orchestrator with git worktree add and git worktree remove; never tie such a worktree to a wrapper agent's lifetime, because harness managed agent worktrees are auto removed when the wrapper agent exits, which deletes the workspace out from under a still running detached job. -- State write permission explicitly in every peer brief. Every Grok mode uses strict. Its upstream `system_read` roots include `/var` and `/tmp` on Linux, and `/private` plus the entire `~/Library` on macOS, so shared temporary content and substantial user application configuration and caches may be readable. Its write roots are the workspace and `~/.grok/sessions` (since 1.0.14 the rest of the Grok home is readable but not writable), always include `/tmp` and `/var/tmp`, and on macOS also include `/private/tmp`, `/private/var/tmp`, and all of `/private/var/folders`. The run's unique private `TMPDIR` is the adapter's preferred and handshake-verified temporary path, not the only writable temporary surface. User toolchains outside the readable roots may therefore be unavailable, and their absence never authorizes a workspace downgrade. Strict requests child process network restriction. Linux enforces it through seccomp, while macOS network blocking is a no-op and a write shell can still reach the network; `--web` controls only the built in web tools. Grok consult briefs and /grok:review add a hard read only tool set limited to file read, list, and search, plus web search and fetch when `--web` is present. They cannot invoke shell commands, tests, git, builds, edits, MCP tools, or subagents through model tools. The sandbox still permits consult `read_file` to reach `~/.grok/auth.json`, config, and sessions because strict keeps the entire Grok home readable, and write terminal commands can receive preserved xAI authentication variables. The companion adds best-effort Read denies for `auth.json`, `mcp_credentials.json`, `config.toml`, `sessions`, `memory`, `logs`, and `debug` through absolute paths and standard `**/.grok` patterns, but raw path variants, symbolic links, and shell or indirect scripts can bypass those tool-level rules. These Read denies govern model tool calls and do not stop upstream's internal first-turn injection for a direct task with explicit `--memory`; injected memory enters the model context sent to xAI. Environment scrubbing and path-pattern denies do not isolate credentials or memory content. Model-facing meta-tool denies do not prove that native MCP servers, plugins, or hooks under `~/.grok` did not start during agent construction or had no side effects; those bridge variables disable compatibility imports, not native Grok configuration. Narrower exposure requires a dedicated sandbox profile, an isolated Grok home, an authentication broker, or upstream path-level authorization. Only briefs asking for repository changes receive the strict write tool set. Before prompt delivery, the companion requires a new `ProfileApplied` event in the incremental delta of the shared `~/.grok/sessions/sandbox-events.jsonl` log (the home-root location that 1.0.13 and earlier wrote is unsupported) that contains this run's private temporary path and matches the canonical workspace, strict profile, enforced state, and requested `restrict_network: true` configuration. Upstream events have no run id or pid, so the unique private path correlates positive evidence; unrelated `ProfileApplied` and `ApplyFailed` records are not attributed, and failure records are only auxiliary diagnostics. The matching event proves profile and configuration application, not actual platform network isolation. A matching warning on the owned stderr stream, handshake timeout, missing or malformed matching evidence, shared-log disappearance or rotation, or any matching field mismatch terminates the process and fails closed; stderr cannot prove successful enforcement. +- State write permission explicitly in every peer brief. Every Grok mode uses strict. Its upstream `system_read` roots include `/var` and `/tmp` on Linux, and `/private` plus the entire `~/Library` on macOS, so shared temporary content and substantial user application configuration and caches may be readable. Its write roots are the workspace and `~/.grok/sessions` (since 1.0.14 the rest of the Grok home is readable but not writable), always include `/tmp` and `/var/tmp`, and on macOS also include `/private/tmp`, `/private/var/tmp`, and all of `/private/var/folders`. The run's unique private `TMPDIR` is the adapter's preferred and handshake-verified temporary path, not the only writable temporary surface. User toolchains outside the readable roots may therefore be unavailable, and their absence never authorizes a workspace downgrade. Strict requests child process network restriction. Linux enforces it through seccomp, while macOS network blocking is a no-op and a write shell can still reach the network; `--web` controls only the built in web tools. Grok consult briefs and /grok:review add a hard read only tool set limited to file read, list, and search, plus web search and fetch when `--web` is present. They cannot invoke shell commands, tests, git, builds, edits, MCP tools, or subagents through model tools. The sandbox still permits consult `read_file` to reach `~/.grok/auth.json`, config, and sessions because strict keeps the entire Grok home readable, and write terminal commands can receive preserved xAI authentication variables. The companion adds best-effort Read denies for `auth.json`, `mcp_credentials.json`, `config.toml`, `sessions`, `memory`, `logs`, and `debug` through absolute paths and standard `**/.grok` patterns, but raw path variants, symbolic links, and shell or indirect scripts can bypass those tool-level rules. These Read denies govern model tool calls and do not stop upstream's internal first-turn injection for a direct task with explicit `--memory`; injected memory enters the model context sent to xAI. Environment scrubbing and path-pattern denies do not isolate credentials or memory content. Model-facing meta-tool denies do not prove that native MCP servers, plugins, or hooks under `~/.grok` did not start during agent construction or had no side effects; the bridge variables gate only loose compatibility imports, not native Grok configuration and not the Claude Code plugins that `~/.claude/settings.json` enables, whose hooks and MCP servers Grok loads at startup unless `~/.grok/config.toml` sets `[claude_compat] imported = true` or lists them in `[plugins] disabled`. Narrower exposure requires a dedicated sandbox profile, an isolated Grok home, an authentication broker, or upstream path-level authorization. Only briefs asking for repository changes receive the strict write tool set. Before prompt delivery, the companion requires a new `ProfileApplied` event in the incremental delta of the shared `~/.grok/sessions/sandbox-events.jsonl` log (the home-root location that 1.0.13 and earlier wrote is unsupported) that contains this run's private temporary path and matches the canonical workspace, strict profile, enforced state, and requested `restrict_network: true` configuration. Upstream events have no run id or pid, so the unique private path correlates positive evidence; unrelated `ProfileApplied` and `ApplyFailed` records are not attributed, and failure records are only auxiliary diagnostics. The matching event proves profile and configuration application, not actual platform network isolation. A matching warning on the owned stderr stream, handshake timeout, missing or malformed matching evidence, shared-log disappearance or rotation, or any matching field mismatch terminates the process and fails closed; stderr cannot prove successful enforcement. - Every peer brief opens with a single header line that names the lane, the effort or model tier whenever the routing row requires one, and the verification command for an implementation or change brief, or explicit coverage or acceptance criteria for a consult, research, or review brief; every automatically routed Grok header also includes `grok-role: `, and every automatically routed Cursor header includes `cursor-role: `. Pass the brief directly in the Agent prompt. Never create a transport prompt file with Write or place a brief under plugin data. The Grok companion must use `--prompt-file /dev/stdin` and pipe the prompt to the child because Grok applies the sandbox before reading a prompt file, which makes an external staged brief unreadable under strict mode. Companion adapters own any private staging required by their transports. Collection review verifies consult, research, and review criteria. A brief missing that header is not ready to dispatch. Claude worker briefs are not peer briefs: their first line is always the `fusion-brief: v1` envelope, and lane or header prose rides inside the envelope after that line. The header line is audit prose: engine model and effort selection happens only through companion options such as `--model` and `--effort` placed before the `--` separator in the direct prompt, and a routing row that names a model tier is satisfied only when the dispatch passes those options. -- Grok review requires the installed headless `--json-schema` contract and consumes `structuredOutput` from the same turn. A schema error or explicit null structured result is a failed review, not permission to trust text or spend a second call. Compatibility parsing of `text` is allowed only when the structured output field is completely absent. Preserve request id, session id, turn count, token usage, per-model usage, cost, partial-cost, and incomplete-usage indicators in job records whenever the CLI reports them. A `model_usage` row reports only input, output, cache-read, and model-call counts plus optional cost, so never synthesize aggregate reasoning or total token channels. `usage_is_incomplete` means the usage ledger may have missed open subagents, usage application, or a drain timeout; present values are observed lower bounds, and both job-total token and cost coverage fail closed to incomplete. Accept only positive complete top-level cost pairs and use ticks as authoritative at 10000000000 ticks per USD. These fields improve observability but never replace independent semantic acceptance. +- Grok review requires the installed headless `--json-schema` contract and consumes `structuredOutput` from the same turn. A schema error or explicit null structured result is a failed review, not permission to trust text or spend a second call. Compatibility parsing of `text` is allowed only when the structured output field is completely absent. Preserve request id, session id, turn count, token usage, per-model usage, cost, partial-cost, and incomplete-usage indicators in job records whenever the CLI reports them. A `model_usage` row reports only input, output, cache-read, cache-creation, and model-call counts plus optional cost, so never synthesize aggregate reasoning or total token channels. `usage_is_incomplete` means the usage ledger may have missed open subagents, usage application, or a drain timeout; present values are observed lower bounds, and both job-total token and cost coverage fail closed to incomplete. Accept only positive complete top-level cost pairs and use ticks as authoritative at 10000000000 ticks per USD. These fields improve observability but never replace independent semantic acceptance. - A future Grok ACP adapter must use one process pool per canonical cwd and sandbox profile. Each daemon starts directly with `grok --cwd --sandbox agent --no-leader stdio`; a global bridge or leader reuse is forbidden because sandbox installation happens once at process start and an existing leader can retain a different cwd or sandbox. ACP permission requests and live tool events improve visibility, but do not weaken the companion's tool, worktree, background, collection, or semantic acceptance boundaries. - Never delegate to codex, grok, or cursor anything touching secrets, credentials, or context that cannot be compressed into a brief. - Runtime guard: in `judgment` posture the hook counts the writes in the current window and attaches one advisory at each multiple of the five write budget, naming the lanes the remaining work could go to; a dispatch or a passing verification command closes the window and clears the advisory, because verified work is finished work. Verification is recognized by passing evidence in the command's output as well as its exit status, so piped and redirected runner invocations count, and absolute paths outside the session workspace (memory files, transport payloads, scratchpad paths) never enter the unverified count while relative paths always count. Advice is never a floor and judgment posture never denies a main loop write: inline sprawl is measured rather than blocked, and it surfaces in the coercion ledger as the deepest unverified window a session reached. In `strict` posture the floors are hard: only a dispatch opens a window, and past the budget the hook permits up to three small Edit calls with replacement content at most 1024 bytes for files already written in the window, plus a zero dispatch relief of ten writes and 16384 bytes that applies to any write tool and any path, while outside those two reliefs every write stays denied. Unreadable guard state fails closed in `strict` and fails open in `judgment`. Two denials hold in both postures, because neither rests on semantics a counter cannot see: a no-op Bash command while Fusion tasks are in flight, which is a silent turn, and a `TaskOutput` or `TaskStop` probe of a worker that already completed and was reaped, which has a result file to read instead. Every user visible guard, lifecycle, and session message ends with a bracketed numeric digest such as `[fusion:1234]`; `/fusion:stats --messages` lists every digest with its meaning and `/fusion:stats --message ` explains one digest and shows its recorded occurrences. `FUSION_INLINE_WRITE_BUDGET`, `FUSION_INLINE_TAIL_MAX_BYTES`, `FUSION_INLINE_TAIL_ALLOWANCE`, `FUSION_INLINE_ZERO_DISPATCH_MAX_BYTES`, and `FUSION_INLINE_ZERO_DISPATCH_WRITES` configure the strict limits, and `FUSION_VERIFICATION_PATTERN` adds a project specific runner to the verification recognizer. The posture dial remains the only control over gate strength. diff --git a/plugins/fusion/rules/troubleshooting.md b/plugins/fusion/rules/troubleshooting.md index 13b8b22..0404090 100644 --- a/plugins/fusion/rules/troubleshooting.md +++ b/plugins/fusion/rules/troubleshooting.md @@ -80,9 +80,9 @@ Grok waits for Bash, monitor, or subagent background work after the first headle Consult, write, and review all require `--sandbox strict`. Before each prompt, the companion creates a unique private `TMPDIR`, snapshots the shared `~/.grok/sessions/sandbox-events.jsonl` log (1.0.14 moved it there and restricted strict writes to that directory; the home-root location is unsupported), launches Grok, and waits for a new `ProfileApplied` event that contains that private path and proves the canonical workspace, strict profile, enforced state, and requested `restrict_network: true` configuration. Upstream records have no run id or pid, so the private path correlates positive evidence; unrelated `ProfileApplied` and `ApplyFailed` records are not attributed, and failure events are only auxiliary diagnostics. The matching event proves profile and configuration application, not actual platform network isolation. A matching owned-stderr warning, handshake timeout, missing or malformed matching evidence, shared-log disappearance or rotation, or a matching field mismatch is a `sandbox` failure that terminates the owned process group. Stderr cannot prove successful enforcement. Never downgrade to workspace. -Strict's upstream `system_read` roots include `/var` and `/tmp` on Linux, and `/private` plus the entire `~/Library` on macOS, so shared temporary content and substantial user application configuration and caches may be readable. Its write roots include the workspace and entire Grok home, always include `/tmp` and `/var/tmp`, and on macOS also include `/private/tmp`, `/private/var/tmp`, and all of `/private/var/folders`. The run-private `TMPDIR` is the adapter's preferred and handshake-verified temporary path, not the only writable temporary surface. Strict is not an operating system read-only profile. Its child network request is enforced through seccomp on Linux, while the current macOS network blocker is a no-op; write shell commands can reach the network on macOS, and `--web` controls only built in web tools. A compiler, package manager, SDK, or other user toolchain outside readable system roots can disappear in write mode. Split or reroute the package, or put the required tool inside an approved readable path; never weaken the sandbox silently. The consult tool deny set and compatibility isolation remain required after successful sandbox application. +Strict's upstream `system_read` roots include `/var` and `/tmp` on Linux, and `/private` plus the entire `~/Library` on macOS, so shared temporary content and substantial user application configuration and caches may be readable. Its write roots are the workspace and `~/.grok/sessions` (since 1.0.14 the rest of the Grok home is readable but not writable), always include `/tmp` and `/var/tmp`, and on macOS also include `/private/tmp`, `/private/var/tmp`, and all of `/private/var/folders`. The run-private `TMPDIR` is the adapter's preferred and handshake-verified temporary path, not the only writable temporary surface. Strict is not an operating system read-only profile. Its child network request is enforced through seccomp on Linux, while the current macOS network blocker is a no-op; write shell commands can reach the network on macOS, and `--web` controls only built in web tools. A compiler, package manager, SDK, or other user toolchain outside readable system roots can disappear in write mode. Split or reroute the package, or put the required tool inside an approved readable path; never weaken the sandbox silently. The consult tool deny set and compatibility isolation remain required after successful sandbox application. -The entire Grok home is a credential and native configuration trust boundary. The sandbox permits consult `read_file` to reach `~/.grok/auth.json`, config, and sessions, while write terminal commands may receive the xAI authentication environment that the scrubber deliberately preserves; since 1.0.14 strict writes only `~/.grok/sessions` inside the home, so those files are readable but not writable from a managed run. The companion layers best-effort Read denies over `auth.json`, `mcp_credentials.json`, `config.toml`, `sessions`, `memory`, `logs`, and `debug` through absolute paths and standard `**/.grok` patterns. Raw path variants, symbolic links, and shell or indirect scripts can bypass those tool-level rules, so environment scrubbing and path-pattern denies do not isolate filesystem credentials. These Read denies govern model tool calls and do not block upstream's internal first-turn memory injection when an ordinary task explicitly enables memory; injected snippets enter the model context sent to xAI. Model-facing `search_tool`, `use_tool`, and MCP denies do not prove that native MCP servers, plugins, or hooks under `~/.grok` did not start during agent construction or had no side effects; those bridge variables disable compatibility imports, not native Grok configuration. Do not delegate untrusted content under the assumption that strict hides Grok credentials, memory, or native startup. Narrower exposure requires a dedicated sandbox profile, an isolated Grok home, an authentication broker, or upstream path-level authorization. +The entire Grok home is a credential and native configuration trust boundary. The sandbox permits consult `read_file` to reach `~/.grok/auth.json`, config, and sessions, while write terminal commands may receive the xAI authentication environment that the scrubber deliberately preserves; since 1.0.14 strict writes only `~/.grok/sessions` inside the home, so those files are readable but not writable from a managed run. The companion layers best-effort Read denies over `auth.json`, `mcp_credentials.json`, `config.toml`, `sessions`, `memory`, `logs`, and `debug` through absolute paths and standard `**/.grok` patterns. Raw path variants, symbolic links, and shell or indirect scripts can bypass those tool-level rules, so environment scrubbing and path-pattern denies do not isolate filesystem credentials. These Read denies govern model tool calls and do not block upstream's internal first-turn memory injection when an ordinary task explicitly enables memory; injected snippets enter the model context sent to xAI. Model-facing `search_tool`, `use_tool`, and MCP denies do not prove that native MCP servers, plugins, or hooks under `~/.grok` did not start during agent construction or had no side effects. The bridge variables gate only loose compatibility imports, not native Grok configuration and not Claude Code plugins: Grok enables the plugins that `~/.claude/settings.json` enables unless `~/.grok/config.toml` sets `[claude_compat] imported = true` or lists them in `[plugins] disabled`, and since 1.0.52 their hooks and MCP servers can start inside a managed run before the first model call. Do not delegate untrusted content under the assumption that strict hides Grok credentials, memory, or native startup. Narrower exposure requires a dedicated sandbox profile, an isolated Grok home, an authentication broker, or upstream path-level authorization. A prompt-file permission failure under strict mode usually means a stale adapter supplied a brief path outside the sandbox. The supported transport is `--prompt-file /dev/stdin` with the complete prompt withheld until the sandbox handshake succeeds. Repair or update the installed plugin rather than widening the sandbox. A zero exit with oversized or malformed headless JSON is `transport`; inspect the bounded tail. Full stdout exists only through a 0600 file descriptor whose directory entry is unlinked immediately after open. Do not revert to a named spool or tail-only parsing. diff --git a/plugins/fusion/skills/doctor/SKILL.md b/plugins/fusion/skills/doctor/SKILL.md index c6e7cec..3a9ef60 100644 --- a/plugins/fusion/skills/doctor/SKILL.md +++ b/plugins/fusion/skills/doctor/SKILL.md @@ -17,7 +17,7 @@ Checks to perform: 1. A set CLAUDE_CODE_SUBAGENT_MODEL can silently override every agent model pin (Claude Code behavior as observed on 2.1.x); flag it as a problem unless the user says it is deliberate. 2. Read ~/.claude/settings.json and report the `model` key. It should be an alias, preferably `best[1m]` (Fable when available, latest Opus otherwise, floats to future releases). Flag any dated full model ID as a rot risk. Also confirm codex@claude-code-fusion, grok@claude-code-fusion, cursor@claude-code-fusion, and fusion@claude-code-fusion appear in enabledPlugins. Treat codex@openai-codex and codex@claude-code-fusion being enabled together as an error because both register the `codex` namespace. When Codex is missing, report that ordinary implementation has lost its primary lane. Bounded isolated work may use Grok under `burst`; work whose load bearing capability is Claude tools or privacy, or that is structurally stranded with a stated `claude-fallback: ` header, uses fusion:claude-worker. Recommend installing codex@claude-code-fusion. -3. Read the frontmatter of every file in ${CLAUDE_PLUGIN_ROOT}/agents/ and report each `model`, `effort`, and `maxTurns` value. Alias pins (opus, sonnet, haiku) float to new releases automatically and are healthy. Full model ID pins are rot risks; fusion:trivial-worker's claude-haiku-4-5 pin is deliberate because the haiku alias can be remapped by ANTHROPIC_DEFAULT_HAIKU_MODEL, so a full ID pins the tier to the intended model regardless of that override, but it must be bumped by hand when a newer cheap tier ships. +3. Read the frontmatter of every file in ${CLAUDE_PLUGIN_ROOT}/agents/ and report each `model`, `effort`, and `maxTurns` value. Alias pins (opus, sonnet, haiku) float to new releases automatically and are healthy. Full model ID pins are rot risks; fusion:trivial-worker's claude-haiku-5-5 pin is deliberate because the haiku alias can be remapped by ANTHROPIC_DEFAULT_HAIKU_MODEL, so a full ID pins the tier to the intended model regardless of that override, but it must be bumped by hand when a newer cheap tier ships. 4. List ~/.claude/agents/ if it exists. A file whose name matches a plugin agent (deep-reasoner, claude-worker, trivial-worker) registers as a duplicate agent alongside the fusion: namespaced one and confuses auto delegation; flag it as a stale leftover to delete unless the user says it is a deliberate local override. 5. Compare the template hash of ~/.claude/rules/orchestration.md against the template hash of ${CLAUDE_PLUGIN_ROOT}/rules/orchestration.md, using the same normalization as rules-sync: replace everything between the `` and `` sentinel lines with the fixed normalized marker before computing SHA-256. Prefer `node --input-type=module` with `hashRulesTemplate` exported by `${CLAUDE_PLUGIN_ROOT}/scripts/lib/rules-template.mjs`; a sed or similar comparison that strips only the sentinel region is also acceptable. Missing live rules or a mismatched template hash means the static routing policy is stale; point at /fusion:setup to install or update it. A user selected live model table may differ from the scored canonical table and is healthy when the template hashes match. 6. Remind the user of the role model: the main Claude session is the only control plane and final judge. Claude workers are native specialists rather than an independent model family, Codex owns ordinary implementation and default deep external review, and Grok keeps the protected `burst`, `independence`, `live-web`, and `large-context` roles. The orchestrator, deep reasoning, mechanical, and trivial Claude tiers are roles bound to alias tiers, not to specific models, so same tier releases (for example Sonnet 5 under the sonnet alias) adopt their role with no configuration change. Only a genuinely new tier above or between the existing rungs requires a decision. @@ -28,9 +28,9 @@ Checks to perform: 11. Check ~/.claude/plugins/data/codex-claude-code-fusion/state, the canonical root for new Codex jobs. If it does not exist, report no recorded Codex job history yet or an uninitialized Codex state path, not necessarily an error. 12. In the live ~/.claude/rules/orchestration.md, collect every concrete model ID named in routing prose outside the `` and `` sentinel region, then compare them with the model IDs in the live capability table. Flag every prose ID without a table row as an orphan. 13. Probe the installed Codex plugin cache's slash-command surface by globbing `~/.claude/plugins/cache/claude-code-fusion/codex/*/skills/*/SKILL.md` for the `/codex:task`, `/codex:rescue`, `/codex:review`, `/codex:adversarial-review`, `/codex:status`, `/codex:result`, `/codex:cancel`, and `/codex:setup` command files and `~/.claude/plugins/cache/claude-code-fusion/codex/*/agents/` for the codex-rescue agent. Report any absent surface as a degraded-capability warning, not an error. -14. Audit the installed Grok runtime and instructions against the actual headless surface, not version assumptions. Consult, write, and review must all use `--sandbox strict`; a workspace downgrade is an error. Every managed run requires `--prompt-file`, `--output-format`, `--sandbox`, `--tools`, `--disallowed-tools`, `--deny`, `--max-turns`, and `--no-auto-update`; consult also requires `--permission-mode` and `--allow`, no-web requires `--disable-web-search`, write requires `--always-approve`, review requires `--json-schema`, ordinary calls require `--no-wait-for-background`. Missing installation, version, or applicable capability readiness is failure kind `setup`; recommend upgrading or repairing the CLI and rerunning `/grok:setup`, never retrying the unchanged task. Consult tools are read, grep, and list plus optional direct web tools. Write tools are `read_file`, `grep`, `list_dir`, `search_replace`, and `run_terminal_cmd`, with Agent always denied; upstream 1.0.0's OpenCode-compatible `write` tool id stays outside the fixed allowlist, and `search_replace` creates files. Every mode denies `search_tool`, `use_tool`, `ask_user_question`, and MCP tools, while ordinary calls also deny Agent. Treat command-pattern denies for common direct grok, claude, and codex invocations as best effort because absolute paths, aliases or functions, and indirect scripts can bypass them. A hard nested-execution boundary requires removing `run_terminal_cmd` or an OS-level executable or network policy. Builder tracing starts only after stdin is consumed, so it cannot prove policy before prompt delivery or every side effect; fallback or unmatched warnings must terminate early, and a successful close without positive `tools allowlist applied` evidence is an error. Confirm the following behavior. The child sets all eighteen `GROK_CLAUDE_*_ENABLED`, `GROK_CURSOR_*_ENABLED`, and `GROK_CODEX_*_ENABLED` bridge variables to false; upstream currently consumes the six Claude and six Cursor cells plus the Codex sessions cell and reserves the other five Codex cells, and the child also pins `GROK_MANAGED_MCPS_ENABLED=false` because that variable has highest precedence over `[managed_mcps]` in `~/.grok/config.toml` and remote settings. Audit the companion's deterministic child environment for 1.0.0: `GROK_WEB_FETCH` is pinned by web mode, `GROK_AUTO_WAKE` and `GROK_MANAGED_MCP_GATEWAY_TOOLS_ENABLED` are pinned false, and `GROK_TEST_VERSION`, `GROK_SANDBOX`, `GROK_LEADER_SOCKET`, `GROK_SUBAGENTS_MAX_DEPTH`, `GROK_WEB_FETCH_PROXY`, and `GROK_WEB_FETCH_ALLOW_LOCAL` are removed. Confirm that it removes inherited `CLAUDE_CODE_*`, `CLAUDE_PLUGIN_*`, `GROK_COMPANION_*`, and `_GROK_CLAUDE_MARKER_OVERRIDE`, broadly scrubs unrelated secrets while preserving xAI authentication, and passes `--no-auto-update`. Confirm `GROK_MEMORY=0` is forced by default, only a direct ordinary task with explicit `--memory` receives `GROK_MEMORY=1`, and Fusion routed briefs, review, and stop gate cannot enable it. Report Grok hard version startup requirements (introduced in 0.2.112, verified through 1.0.41; exit 1 regardless of `--no-auto-update`, classified `setup`) as the residual execution and network boundary. The companion's own tested interval is `[1.0.14, 1.0.42)`: below it managed runs fail closed as `setup`, at or past it `/grok:setup` reports the build as newer than the tested interval and a verification pass is advised. -15. Audit native Grok configuration as a user-controlled trust boundary. The `search_tool`, `use_tool`, and MCP denies block model invocation but do not prove that native MCP servers, plugins, or hooks under `~/.grok` were not started during agent construction or had no side effects; those bridge variables disable Codex compatibility imports, not native Grok configuration. Run `grok inspect --json` and review the reported `permissions`, `hooks`, `plugins`, `agents`, `mcpServers`, and `externalCompat` surfaces as the native configuration exposure check that the SECURITY threat model otherwise cannot verify. Use scoped `rg` queries that report configured surface names without printing credentials or full configuration, and label any native startup surface as an explicit trust-boundary notice rather than claiming MCP side effects are disabled. The Grok home is readable under strict, but since 1.0.14 strict writes only `~/.grok/sessions` inside it, and the sandbox permits consult `read_file` to reach `~/.grok/auth.json`, config, and sessions while write terminal commands may receive preserved xAI authentication variables. Confirm the companion's best-effort Read denies cover `auth.json`, `mcp_credentials.json`, `config.toml`, `sessions`, `memory`, `logs`, and `debug` with both absolute paths and standard `**/.grok` patterns. Report that raw path variants, symbolic links, and shell or indirect scripts can bypass those tool-level rules, so environment scrubbing and pattern denies do not isolate filesystem credentials. When memory is explicitly enabled, upstream's internal first-turn memory injection is not a model `read_file` call and can place selected memory in the model context sent to xAI despite those Read denies. Narrower exposure requires a dedicated sandbox profile, an isolated Grok home, an authentication broker, or upstream path-level authorization. -16. Audit Grok sandbox, transport, output, metrics, session, worktree, collection, continuity, history, and ACP evidence. Strict's upstream `system_read` roots include `/var` and `/tmp` on Linux, and `/private` plus the entire `~/Library` on macOS, so shared temporary content and substantial user application configuration and caches may be readable. Its write roots are the workspace and `~/.grok/sessions` (since 1.0.14 the rest of the Grok home is readable but not writable), always include `/tmp` and `/var/tmp`, and on macOS also include `/private/tmp`, `/private/var/tmp`, and all of `/private/var/folders`; the unique private `TMPDIR` is preferred and handshake-verified, not the only writable temporary surface. Before prompt delivery, the adapter must validate a new `ProfileApplied` event from the incremental `~/.grok/sessions/sandbox-events.jsonl` delta (the home-root location that 1.0.13 and earlier wrote is unsupported) that contains this run's private path and matches the canonical workspace, strict profile, enforced state, and requested `restrict_network: true` configuration. Upstream events have no run id or pid, so unrelated `ProfileApplied` and `ApplyFailed` records are not attributable to this run; failure events are auxiliary only. A matching owned-stderr warning, handshake timeout, missing or malformed matching evidence, shared-log rotation or disappearance, or matching field mismatch fails closed. The matching event proves configuration application, not actual network isolation: Linux uses seccomp, while macOS network blocking is a no-op and write shells may reach the network. Every prompt uses `--prompt-file /dev/stdin`. Stdout uses a 0600 file unlinked immediately after open and parsed through its descriptor under the total limit. An upstream `model_usage` row contains only input, output, cache-read, and model-call counts plus optional cost, never synthetic reasoning or total tokens. `usage_is_incomplete` makes reported values observed lower bounds and both job-total token and cost coverage incomplete. Exact cost otherwise requires a positive complete USD and tick pair, with ticks authoritative at 10000000000 per USD. Short-cwd metadata uses only the exact encoded path; long-cwd lookup requires an exact decoded or `.cwd` match and never selects a sole cross-cwd candidate. Review requires one `--json-schema` call; an error or explicit null structured result fails, and only a completely absent field permits local text parsing. Continuity must default to `manual`; optional `claude-session` affinity may select only a completed ordinary task from the same Claude session, exact resolved cwd, mode, strict profile, and memory boundary. Fusion routed briefs must not receive automatic affinity, while explicit resume remains limited to compatible terminal task records. `/grok:history` must expose only the safe companion metadata projection, default to the current workspace, and never read native session contents, briefs, results, or logs. Treat native sessions, companion history, and cross-session memory as three separate stores. Native session cleanup defaults to 30 days and accepts only a positive `[storage] cleanup_ttl_days`, with `0` falling back to 30; the companion ledger has no promised automatic garbage collection. Fusion creates real worktrees. Upstream ships ACP today, but the companion has not adopted it and continues per-call invocation; any future companion adoption must pool by canonical cwd plus sandbox profile with `grok --cwd --sandbox agent --no-leader stdio`. A collector timeout, dead job, or status error reports the Fusion task id, peer job id, `uncollected`, and exact result command. Successful Grok collection remains unverified until `/fusion:stats --record =` records the main-loop judgment. +14. Audit the installed Grok runtime and instructions against the actual headless surface, not version assumptions. Consult, write, and review must all use `--sandbox strict`; a workspace downgrade is an error. Every managed run requires `--prompt-file`, `--output-format`, `--sandbox`, `--tools`, `--disallowed-tools`, `--deny`, `--max-turns`, and `--no-auto-update`; consult also requires `--permission-mode` and `--allow`, no-web requires `--disable-web-search`, write requires `--always-approve`, review requires `--json-schema`, ordinary calls require `--no-wait-for-background`. Missing installation, version, or applicable capability readiness is failure kind `setup`; recommend upgrading or repairing the CLI and rerunning `/grok:setup`, never retrying the unchanged task. Consult tools are read, grep, and list plus optional direct web tools. Write tools are `read_file`, `grep`, `list_dir`, `search_replace`, and `run_terminal_cmd`, with Agent always denied; upstream 1.0.0's OpenCode-compatible `write` tool id stays outside the fixed allowlist, and `search_replace` creates files. Every mode denies `search_tool`, `use_tool`, `ask_user_question`, and MCP tools, while ordinary calls also deny Agent. Treat command-pattern denies for common direct grok, claude, and codex invocations as best effort because absolute paths, aliases or functions, and indirect scripts can bypass them. A hard nested-execution boundary requires removing `run_terminal_cmd` or an OS-level executable or network policy. Builder tracing starts only after stdin is consumed, so it cannot prove policy before prompt delivery or every side effect; fallback or unmatched warnings must terminate early, and a successful close without positive `tools allowlist applied` evidence is an error. Confirm the following behavior. The child sets all eighteen `GROK_CLAUDE_*_ENABLED`, `GROK_CURSOR_*_ENABLED`, and `GROK_CODEX_*_ENABLED` bridge variables to false; upstream currently consumes the six Claude and six Cursor cells plus the Codex sessions cell and reserves the other five Codex cells, and the child also pins `GROK_MANAGED_MCPS_ENABLED=false` because that variable has highest precedence over `[managed_mcps]` in `~/.grok/config.toml` and remote settings. Audit the companion's deterministic child environment for 1.0.0: `GROK_WEB_FETCH` is pinned by web mode, `GROK_AUTO_WAKE`, `GROK_MANAGED_MCP_GATEWAY_TOOLS_ENABLED`, `GROK_TURN_SUMMARY`, and `GROK_FILE_ACCELERATION` are pinned false, and `GROK_TEST_VERSION`, `GROK_SANDBOX`, `GROK_LEADER_SOCKET`, `GROK_SUBAGENTS_MAX_DEPTH`, `GROK_WEB_FETCH_PROXY`, `GROK_WEB_FETCH_ALLOW_LOCAL`, `GROK_SANDBOX_MODE`, `GROK_ARTIFACTS_BIND_REMOUNT`, `GROK_FILE_ACCELERATION_ROUTES`, `GROK_CURSOR_WORKER_API_BASE`, and `GROK_CURSOR_WORKER_COMMAND` are removed. Confirm that it removes inherited `CLAUDE_CODE_*`, `CLAUDE_PLUGIN_*`, `GROK_COMPANION_*`, and `_GROK_CLAUDE_MARKER_OVERRIDE`, broadly scrubs unrelated secrets while preserving xAI authentication, and passes `--no-auto-update`. Confirm `GROK_MEMORY=0` is forced by default, only a direct ordinary task with explicit `--memory` receives `GROK_MEMORY=1`, and Fusion routed briefs, review, and stop gate cannot enable it. Report Grok hard version startup requirements (introduced in 0.2.112, verified through 1.0.41; exit 1 regardless of `--no-auto-update`, classified `setup`) as the residual execution and network boundary. The companion's own tested interval is `[1.0.14, 1.0.42)`: below it managed runs fail closed as `setup`, at or past it `/grok:setup` reports the build as newer than the tested interval and a verification pass is advised. +15. Audit native Grok configuration as a user-controlled trust boundary. The `search_tool`, `use_tool`, and MCP denies block model invocation but do not prove that native MCP servers, plugins, or hooks under `~/.grok` were not started during agent construction or had no side effects, and the bridge variables gate only loose compatibility imports, not native Grok configuration and not Claude Code plugins. Report every plugin that `~/.claude/settings.json` enables (a scoped `node` read of `enabledPlugins` only) as loaded into managed Grok runs, hooks and MCP servers included, unless `~/.grok/config.toml` sets `[claude_compat] imported = true` or lists it in `[plugins] disabled` (scoped `rg` for those two keys only). Run `grok inspect --json` and review the reported `permissions`, `hooks`, `plugins`, `agents`, `mcpServers`, and `externalCompat` surfaces as the native configuration exposure check that the SECURITY threat model otherwise cannot verify. Use scoped `rg` queries that report configured surface names without printing credentials or full configuration, and label any native startup surface as an explicit trust-boundary notice rather than claiming MCP side effects are disabled. The Grok home is readable under strict, but since 1.0.14 strict writes only `~/.grok/sessions` inside it, and the sandbox permits consult `read_file` to reach `~/.grok/auth.json`, config, and sessions while write terminal commands may receive preserved xAI authentication variables. Confirm the companion's best-effort Read denies cover `auth.json`, `mcp_credentials.json`, `config.toml`, `sessions`, `memory`, `logs`, and `debug` with both absolute paths and standard `**/.grok` patterns. Report that raw path variants, symbolic links, and shell or indirect scripts can bypass those tool-level rules, so environment scrubbing and pattern denies do not isolate filesystem credentials. When memory is explicitly enabled, upstream's internal first-turn memory injection is not a model `read_file` call and can place selected memory in the model context sent to xAI despite those Read denies. Narrower exposure requires a dedicated sandbox profile, an isolated Grok home, an authentication broker, or upstream path-level authorization. +16. Audit Grok sandbox, transport, output, metrics, session, worktree, collection, continuity, history, and ACP evidence. Strict's upstream `system_read` roots include `/var` and `/tmp` on Linux, and `/private` plus the entire `~/Library` on macOS, so shared temporary content and substantial user application configuration and caches may be readable. Its write roots are the workspace and `~/.grok/sessions` (since 1.0.14 the rest of the Grok home is readable but not writable), always include `/tmp` and `/var/tmp`, and on macOS also include `/private/tmp`, `/private/var/tmp`, and all of `/private/var/folders`; the unique private `TMPDIR` is preferred and handshake-verified, not the only writable temporary surface. Before prompt delivery, the adapter must validate a new `ProfileApplied` event from the incremental `~/.grok/sessions/sandbox-events.jsonl` delta (the home-root location that 1.0.13 and earlier wrote is unsupported) that contains this run's private path and matches the canonical workspace, strict profile, enforced state, and requested `restrict_network: true` configuration. Upstream events have no run id or pid, so unrelated `ProfileApplied` and `ApplyFailed` records are not attributable to this run; failure events are auxiliary only. A matching owned-stderr warning, handshake timeout, missing or malformed matching evidence, shared-log rotation or disappearance, or matching field mismatch fails closed. The matching event proves configuration application, not actual network isolation: Linux uses seccomp, while macOS network blocking is a no-op and write shells may reach the network. Every prompt uses `--prompt-file /dev/stdin`. Stdout uses a 0600 file unlinked immediately after open and parsed through its descriptor under the total limit. An upstream `model_usage` row contains only input, output, cache-read, cache-creation, and model-call counts plus optional cost, never synthetic reasoning or total tokens. `usage_is_incomplete` makes reported values observed lower bounds and both job-total token and cost coverage incomplete. Exact cost otherwise requires a positive complete USD and tick pair, with ticks authoritative at 10000000000 per USD. Short-cwd metadata uses only the exact encoded path; long-cwd lookup requires an exact decoded or `.cwd` match and never selects a sole cross-cwd candidate. Review requires one `--json-schema` call; an error or explicit null structured result fails, and only a completely absent field permits local text parsing. Continuity must default to `manual`; optional `claude-session` affinity may select only a completed ordinary task from the same Claude session, exact resolved cwd, mode, strict profile, and memory boundary. Fusion routed briefs must not receive automatic affinity, while explicit resume remains limited to compatible terminal task records. `/grok:history` must expose only the safe companion metadata projection, default to the current workspace, and never read native session contents, briefs, results, or logs. Treat native sessions, companion history, and cross-session memory as three separate stores. Native session cleanup defaults to 30 days and accepts only a positive `[storage] cleanup_ttl_days`, with `0` falling back to 30; the companion ledger has no promised automatic garbage collection. Fusion creates real worktrees. Upstream ships ACP today, but the companion has not adopted it and continues per-call invocation; any future companion adoption must pool by canonical cwd plus sandbox profile with `grok --cwd --sandbox agent --no-leader stdio`. A collector timeout, dead job, or status error reports the Fusion task id, peer job id, `uncollected`, and exact result command. Successful Grok collection remains unverified until `/fusion:stats --record =` records the main-loop judgment. 17. Audit the Cursor isolation evidence. For the newest few records under `~/.claude/plugins/data/cursor-claude-code-fusion/state/*/jobs/`, confirm each carries a `homeDir` under that plugin's `runs/` directory, that the home's `.cursor/plugins` and `.cursor/skills-cursor` directories are empty and not writable, that `.cursor/cli-config.json` has `approvalMode` `allowlist` and `sandbox.mode` `enabled`, and that read only modes deny `Write(**)`. A record without a private home, a populated plugins directory, or a widened config means a run escaped the companion's isolation recipe; report it as a problem. Report records whose `warnings` name project hook files as informational, with the repository they came from. Read `selectedModel.modelId` from `~/.cursor/cli-config.json` with a scoped rg query and report it as the model that direct `/cursor:*` commands fall back to; Fusion routed dispatches always carry the cursor lane defaults instead. 18. Run `codex --version`, `grok --version`, and `cursor-agent --version` (never the bare `agent` name, which other CLIs also install) and compare them against the verified contract versions in `${CLAUDE_PLUGIN_ROOT}/verified-versions.json`. An installed version older than the verified one is a compatibility warning. A newer one is an informational drift notice that the companion contracts were verified against the older release and behavior may have moved. Neither is an error by itself because a companion call failing with failure kind `setup` is the enforcement signal. Preserve the `/grok:setup` `compatibility` line for the tested interval `[1.0.14, 1.0.42)`, where `ready` also requires the installed build to be inside the interval and a newer build renders a verification-pass next step. Job records carry `grokVersion`, so the breaker advisory names the installed build next to the verified pin when they differ. Multiple Codex generations sharing one `CODEX_HOME` (for example an IDE or desktop app bundling a newer Codex) are worth flagging when the versions straddle the verified release. diff --git a/tests/rules-sync.test.mjs b/tests/rules-sync.test.mjs index 771c53c..49bf421 100644 --- a/tests/rules-sync.test.mjs +++ b/tests/rules-sync.test.mjs @@ -351,7 +351,8 @@ test("Grok rules document source verified headless boundaries", () => { assert.match(rules, /not the only writable temporary surface/); assert.match(rules, /sandbox still permits consult `read_file` to reach `~\/\.grok\/auth\.json`/i); assert.match(rules, /native MCP servers, plugins, or hooks/); - assert.match(rules, /bridge variables disable compatibility imports, not native Grok configuration/); + assert.match(rules, /bridge variables gate only loose compatibility imports, not native Grok configuration and not the Claude Code plugins that `~\/\.claude\/settings\.json` enables/); + assert.match(rules, /`\[claude_compat\] imported = true` or lists them in `\[plugins\] disabled`/); for (const sentence of [ freshSessionSentence, volumeTierSentence, @@ -414,7 +415,7 @@ test("Grok rules document source verified headless boundaries", () => { assert.match(contract, /Upstream ships ACP today/); assert.match(contract, /The companion has not adopted ACP yet and continues per-call invocation/); assert.match(codexContract, /failureKind: "setup".*below the tested minimum 0\.154\.0/); - assert.match(codexContract, /The tested interval runs from 0\.154\.0 up to but excluding 0\.158\.0; versions below 0\.154\.0 fail task and review preflight with a setup error; versions at or above 0\.158\.0 keep running task and review preflight while `\/codex:setup` reports not ready with exit code 1 as the compatibility advisory/); + assert.match(codexContract, /The tested interval runs from 0\.154\.0 up to but excluding 0\.163\.0; versions below 0\.154\.0 fail task and review preflight with a setup error; versions at or above 0\.163\.0 keep running task and review preflight while `\/codex:setup` reports not ready with exit code 1 as the compatibility advisory/); assert.match(codexContract, /Codex configuration parse failures under `--strict-config`.*failureKind: "process"/); assert.match(sharedContract, /The Grok instance adds `sandbox`[^\n]*`transport`[^\n]*`policy`[^\n]*`turn_limit`[^\n]*and `network`/); assert.match(sharedContract, /`setup`: The installed CLI version or installation lacks a required adapter capability and fails capability preflight/); @@ -433,7 +434,8 @@ test("Grok rules document source verified headless boundaries", () => { assert.match(doctor, /Bash\(grok inspect:\*\)/); assert.match(doctor, /Run `grok inspect --json` and review the reported `permissions`, `hooks`, `plugins`, `agents`, `mcpServers`, and `externalCompat` surfaces/); assert.match(doctor, /successful Grok collection remains unverified until `\/fusion:stats --record =`/i); - assert.match(doctor, /`GROK_WEB_FETCH` is pinned by web mode, `GROK_AUTO_WAKE` and `GROK_MANAGED_MCP_GATEWAY_TOOLS_ENABLED` are pinned false/); + assert.match(doctor, /`GROK_WEB_FETCH` is pinned by web mode, `GROK_AUTO_WAKE`, `GROK_MANAGED_MCP_GATEWAY_TOOLS_ENABLED`, `GROK_TURN_SUMMARY`, and `GROK_FILE_ACCELERATION` are pinned false/); + assert.match(doctor, /`GROK_SANDBOX_MODE`, `GROK_ARTIFACTS_BIND_REMOUNT`, `GROK_FILE_ACCELERATION_ROUTES`, `GROK_CURSOR_WORKER_API_BASE`, and `GROK_CURSOR_WORKER_COMMAND` are removed/); for (const text of [readme, security]) { assert.match(text, /introduced in 0\.2\.112, verified through 1\.0\.41/); assert.match(text, /file that is unlinked (?:immediately after open|as soon as it is opened)/); From 2ed0841cefe068565e8135357d3fd9f3da0df0ce Mon Sep 17 00:00:00 2001 From: Harry Yep Date: Tue, 13 Oct 2026 17:25:49 +0800 Subject: [PATCH 7/8] chore: release 0.0.64 --- .claude-plugin/marketplace.json | 10 +++++----- CHANGELOG.md | 9 +++++++++ plugins/codex/.claude-plugin/plugin.json | 2 +- plugins/cursor/.claude-plugin/plugin.json | 2 +- plugins/fusion/.claude-plugin/plugin.json | 2 +- plugins/grok/.claude-plugin/plugin.json | 2 +- 6 files changed, 18 insertions(+), 9 deletions(-) diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 5802be4..d43bfe6 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -6,14 +6,14 @@ "email": "hi@okis.dev" }, "description": "Multi-model orchestration marketplace for Claude Code.", - "version": "0.0.63", + "version": "0.0.64", "plugins": [ { "name": "grok", "source": "./plugins/grok", "displayName": "Grok Companion", "description": "Local Grok CLI delegation: task, review, resumable history, best-of-n tournaments, background jobs, stats, and setup health checks.", - "version": "0.0.63", + "version": "0.0.64", "author": { "name": "Harry Yep" }, @@ -34,7 +34,7 @@ "source": "./plugins/codex", "displayName": "Codex Companion", "description": "First party local Codex CLI delegation for tasks, reviews, resumable threads, and durable background jobs.", - "version": "0.0.63", + "version": "0.0.64", "author": { "name": "Harry Yep" }, @@ -55,7 +55,7 @@ "source": "./plugins/cursor", "displayName": "Cursor Companion", "description": "Local Cursor Agent CLI delegation: multi family consults, Composer implementation in owned worktrees, review, resumable history, background jobs, and setup health checks.", - "version": "0.0.63", + "version": "0.0.64", "author": { "name": "Harry Yep" }, @@ -76,7 +76,7 @@ "source": "./plugins/fusion", "displayName": "Fusion Orchestrator", "description": "Multi-model orchestration: tier agents, routing rules, blind panel, ultra fleet, model config, and drift doctor.", - "version": "0.0.63", + "version": "0.0.64", "author": { "name": "Harry Yep" }, diff --git a/CHANGELOG.md b/CHANGELOG.md index 4a419b8..5eda18d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,14 @@ # changelog +## 0.0.64 + +- the codex companion is verified on codex-cli 0.162.1 (#72). the source diff from rust-v0.157.1 changes none of the flags, jsonl events, resume, output schema, review path, login status, models cache, or rollout layout the companion relies on, and setup, `--output-schema`, thread resume, and native review pass live on 0.162.1. the tested interval moves to `[0.154.0, 0.163.0)` and `verified-versions.json` to 0.162.1. `Flex capacity unavailable.` now classifies as `rate_limited`, and the content filter and stock cybersecurity and biological risk refusals classify as `refused` instead of `error`. +- the grok child env covers the 1.0.52 surfaces (#72): `GROK_TURN_SUMMARY=false` stops the extra grok-4.5 summary call per turn, `GROK_FILE_ACCELERATION=false` is pinned, and inherited `GROK_SANDBOX_MODE`, `GROK_ARTIFACTS_BIND_REMOUNT`, `GROK_FILE_ACCELERATION_ROUTES`, `GROK_CURSOR_WORKER_API_BASE`, and `GROK_CURSOR_WORKER_COMMAND` no longer reach managed runs. all eighteen bridge pins are confirmed effective on the installed 1.0.52. the grok tested interval stays `[1.0.14, 1.0.42)` until a live pass (#75), which an exhausted grok balance and the OrbStack write refusal block on both hosts today. +- the docs stop claiming the bridge variables cover every compatibility import. grok enables the claude code plugins that `~/.claude/settings.json` enables, and since 1.0.52 their hooks and mcp servers can start inside a managed run unless `~/.grok/config.toml` sets `[claude_compat] imported = true` or lists them in `[plugins] disabled`. security, the readme, the grok contract, the rules, and the doctor now say so, and also fix the stale `[1.0.14, 1.0.31)` interval, the "entire grok home" write root, and the `modelUsage` field names. +- `bench/lanes/` measures one routing lane at a time (#70). `run.mjs` sends a task brief through one companion with an explicit model and effort, verifies outside the workspace, and appends a jsonl record; `summarize.mjs` groups the records. the first snapshot, `bench/results/lanes-2026-10-13/`, runs composer-2.5, composer-2.5-fast, and gpt-6-luna on T05, T06, and T09, plus four blind consults through muse-spark-1.3, gemini-3.8-flash, kimi-k3, glm-5.2, and deep-reasoner. composer stays co-equal with grok burst (0 of 6 on T06, where luna passes 9 of 9), muse-spark-1.3 stays the diversity default, kimi-k3 rises to intelligence 4, and the five cursor rows trade their provisional notes for the measurements. +- `fusion:trivial-worker` pins `claude-haiku-5-5` (#73). +- the consults surfaced a latent double removal in the T01 reference solution, tracked in #74. + ## 0.0.63 - cursor joins as the third peer engine. the new `cursor` plugin drives the cursor agent cli (`cursor-agent`, build 2026.10.01-e373342) headlessly with consult, write, and review modes, resumable chats, background jobs, history, a model catalog, and setup. fusion routes it under two protected roles: `diversity`, a read only consult on a model family no other lane serves (google, meta, moonshot, zhipu, cursor composer), and `burst`, composer implementation inside orchestrator owned worktrees. it never runs auto, never serves the claude family, and only stands in for another family's weights when that family's native lane is down on quota. diff --git a/plugins/codex/.claude-plugin/plugin.json b/plugins/codex/.claude-plugin/plugin.json index 8b6df7a..59b562f 100644 --- a/plugins/codex/.claude-plugin/plugin.json +++ b/plugins/codex/.claude-plugin/plugin.json @@ -2,7 +2,7 @@ "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", "name": "codex", "displayName": "Codex Companion", - "version": "0.0.63", + "version": "0.0.64", "description": "First party local Codex CLI delegation for tasks, reviews, resumable threads, and durable background jobs.", "author": { "name": "Harry Yep" diff --git a/plugins/cursor/.claude-plugin/plugin.json b/plugins/cursor/.claude-plugin/plugin.json index 561b1f7..7a4016f 100644 --- a/plugins/cursor/.claude-plugin/plugin.json +++ b/plugins/cursor/.claude-plugin/plugin.json @@ -2,7 +2,7 @@ "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", "name": "cursor", "displayName": "Cursor Companion", - "version": "0.0.63", + "version": "0.0.64", "description": "Local Cursor Agent CLI delegation: multi family consults, Composer implementation in owned worktrees, review, resumable history, background jobs, and setup health checks.", "author": { "name": "Harry Yep" diff --git a/plugins/fusion/.claude-plugin/plugin.json b/plugins/fusion/.claude-plugin/plugin.json index d870d41..b642ac1 100644 --- a/plugins/fusion/.claude-plugin/plugin.json +++ b/plugins/fusion/.claude-plugin/plugin.json @@ -2,7 +2,7 @@ "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", "name": "fusion", "displayName": "Fusion Orchestrator", - "version": "0.0.63", + "version": "0.0.64", "description": "Multi-model orchestration: tier agents, routing rules, blind panel, ultra fleet, model config, and drift doctor.", "author": { "name": "Harry Yep" diff --git a/plugins/grok/.claude-plugin/plugin.json b/plugins/grok/.claude-plugin/plugin.json index d573567..7639b13 100644 --- a/plugins/grok/.claude-plugin/plugin.json +++ b/plugins/grok/.claude-plugin/plugin.json @@ -2,7 +2,7 @@ "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", "name": "grok", "displayName": "Grok Companion", - "version": "0.0.63", + "version": "0.0.64", "description": "Local Grok CLI delegation: task, review, resumable history, background jobs, stats, and setup health checks.", "author": { "name": "Harry Yep" From a6e5399a8a22e265755bec45876b7a4e7f00fcc6 Mon Sep 17 00:00:00 2001 From: Harry Yep Date: Tue, 13 Oct 2026 17:28:57 +0800 Subject: [PATCH 8/8] docs: note that the claude import marker also stops the settings env import --- SECURITY.md | 2 +- docs/grok-contract.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/SECURITY.md b/SECURITY.md index ad061ac..4ae6187 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -8,7 +8,7 @@ Before prompt delivery, the companion requires a new `ProfileApplied` event in t The child sets all eighteen `GROK_CLAUDE_*_ENABLED`, `GROK_CURSOR_*_ENABLED`, and `GROK_CODEX_*_ENABLED` bridge variables to false; upstream currently consumes the six Claude and six Cursor cells plus the Codex sessions cell and reserves the other five Codex cells, and the child also pins `GROK_MANAGED_MCPS_ENABLED=false` because that variable has highest precedence over `[managed_mcps]` in `~/.grok/config.toml` and remote settings. It also pins `GROK_MANAGED_MCP_GATEWAY_TOOLS_ENABLED=false`, `GROK_AUTO_WAKE=false`, `GROK_TURN_SUMMARY=false` (the turn summary feature defaults on and makes an extra model call per turn), `GROK_FILE_ACCELERATION=false`, and `GROK_WEB_FETCH=1` with `--web` or `GROK_WEB_FETCH=0` without it. It disables subagents, disables automatic updates through `--no-auto-update`, removes inherited Claude session and plugin variables plus `_GROK_CLAUDE_MARKER_OVERRIDE`, `GROK_TEST_VERSION`, `GROK_SANDBOX`, `GROK_LEADER_SOCKET`, `GROK_SUBAGENTS_MAX_DEPTH`, `GROK_WEB_FETCH_PROXY`, `GROK_WEB_FETCH_ALLOW_LOCAL`, `GROK_SANDBOX_MODE`, `GROK_ARTIFACTS_BIND_REMOUNT`, `GROK_FILE_ACCELERATION_ROUTES`, `GROK_CURSOR_WORKER_API_BASE`, and `GROK_CURSOR_WORKER_COMMAND`, and broadly scrubs secret-bearing variables while preserving required xAI authentication and organization version-policy variables. Removing the web fetch proxy and local variables does not remove a proxy or local allowance configured in `~/.grok/config.toml`, which takes precedence over the environment. Grok enforces hard version requirements at startup regardless of `--no-auto-update` (introduced in 0.2.112, verified through 1.0.41; organization message, exit 1 before any envelope), which the companion classifies as `setup`. The companion's own tested interval is `[1.0.14, 1.0.42)`: below it managed runs fail closed as `setup`, at or past it `/grok:setup` reports the build as newer than the tested interval and a verification pass is advised; job records carry `grokVersion` so a breaker advisory can name the build that failed. Memory is force-disabled unless the user explicitly passes `--memory` to an ordinary task. Review and stop gate always keep memory off. A memory-enabled task allows upstream to inject selected global or workspace memory into the model context sent to xAI and to update memory according to the user's Grok configuration; since 1.0.52 a memory-enabled turn can also schedule memory maintenance after it ends, which can make further model calls. Upstream automatic saving usually requires at least three real user prompts in the same resumed session, so one Fusion task usually only reads existing memory and does not guarantee that new memory is saved. Continuity defaults to manual. Optional `claude-session` affinity resumes only a compatible completed ordinary direct task from the same Claude session, exact resolved working directory, mode, strict profile, and memory boundary. Fusion routed briefs do not receive automatic affinity and stay fresh unless explicitly resumed; `--fresh` disables affinity for one task. Write includes terminal commands, so treat it as giving Grok your own shell inside the strict workspace view and run it on a clean branch or an orchestrator-created disposable worktree. Command-pattern denies for common direct grok, claude, and codex calls do not cover absolute paths, aliases or functions, or indirect scripts; a hard nested-execution boundary requires removing `run_terminal_cmd` or applying an OS-level executable or network policy. Headless stdout is captured through a 0600 file that is unlinked immediately after open. `grok --version` now prints `grok 1.0.30 ()`; the companion records the raw string and never parses it. Grok 1.0.0 introduces inline agent definitions (`--agent` and `--agents`), `--rules`, and `--system-prompt-override`, which the companion never passes, so managed runs are unaffected. A resumable Grok task keeps its strict profile, mode, and memory boundary for its lifetime, so incompatible resumes fail closed. Job briefs and logs are stored in plain text under `~/.claude/plugins/data/grok-claude-code-fusion/`. The routing policy, panel blindness, and circuit breaker are prompt level conventions, not runtime enforcement; see the Safety model section of the README for the full split between what code enforces and what prompts request. -Upstream strict's `system_read` roots include `/var` and `/tmp` on Linux, and `/private` plus the entire `~/Library` on macOS. Shared temporary content and substantial user application configuration and caches may therefore be readable. Its write roots are the workspace and `~/.grok/sessions` (since 1.0.14 the rest of the Grok home is readable but not writable), always include `/tmp` and `/var/tmp`, and on macOS also include `/private/tmp`, `/private/var/tmp`, and all of `/private/var/folders`. The adapter's unique private `TMPDIR` is preferred and handshake-verified, not the only writable temporary surface. On Linux, a strict profile with deny paths re-executes under bubblewrap and refuses to start when `bwrap` is missing or unusable; macOS has no equivalent hard enforcement and relies on Seatbelt application plus best-effort tool denies. The sandbox permits consult `read_file` to reach `~/.grok/auth.json`, config, and sessions because strict keeps the Grok home readable, while write terminal commands can receive preserved xAI authentication variables. The companion applies best-effort Read denies to `auth.json`, `mcp_credentials.json`, `config.toml`, `sessions`, `memory`, `logs`, and `debug` under the Grok home with absolute paths and standard `**/.grok` patterns, but raw path variants, symbolic links, and shell or indirect scripts can bypass those tool-level rules. These Read denies apply to model tool calls. They do not prevent upstream's internal first-turn memory search and injection when `--memory` is enabled, so memory snippets can enter the xAI request even though direct model reads of `~/.grok/memory/**` are denied. Environment scrubbing and path-pattern denies do not isolate credentials or memory content. Model-facing MCP and meta-tool denies do not prove that native MCP servers, plugins, or hooks configured under `~/.grok` did not start during agent construction or had no side effects. The bridge variables gate only Grok's loose compatibility imports (skills, rules, agents, hooks, MCP servers, and sessions read from other tools' configuration), not native Grok configuration and not Claude Code plugins: unless `~/.grok/config.toml` sets `[claude_compat] imported = true`, Grok enables every plugin that `~/.claude/settings.json` enables, and since 1.0.52 an enabled and trusted plugin's hooks join the session at startup, so Claude Code plugin hooks and MCP servers can run inside a managed run before the first model call. Listing those plugins in `[plugins] disabled`, or setting `[claude_compat] imported = true`, keeps them out. Grok also passes the `env` block of `~/.claude/settings.json`, trusted project settings, and `.envrc` into tool subprocesses, so scrubbing the child environment does not keep those values out of write-mode shells. Startup can create directories and trust-boundary files inside the Grok home before confinement applies, so the sessions-only write grant describes the enforced phase. Narrower exposure requires a dedicated sandbox profile, an isolated Grok home, an authentication broker, or upstream path-level authorization. +Upstream strict's `system_read` roots include `/var` and `/tmp` on Linux, and `/private` plus the entire `~/Library` on macOS. Shared temporary content and substantial user application configuration and caches may therefore be readable. Its write roots are the workspace and `~/.grok/sessions` (since 1.0.14 the rest of the Grok home is readable but not writable), always include `/tmp` and `/var/tmp`, and on macOS also include `/private/tmp`, `/private/var/tmp`, and all of `/private/var/folders`. The adapter's unique private `TMPDIR` is preferred and handshake-verified, not the only writable temporary surface. On Linux, a strict profile with deny paths re-executes under bubblewrap and refuses to start when `bwrap` is missing or unusable; macOS has no equivalent hard enforcement and relies on Seatbelt application plus best-effort tool denies. The sandbox permits consult `read_file` to reach `~/.grok/auth.json`, config, and sessions because strict keeps the Grok home readable, while write terminal commands can receive preserved xAI authentication variables. The companion applies best-effort Read denies to `auth.json`, `mcp_credentials.json`, `config.toml`, `sessions`, `memory`, `logs`, and `debug` under the Grok home with absolute paths and standard `**/.grok` patterns, but raw path variants, symbolic links, and shell or indirect scripts can bypass those tool-level rules. These Read denies apply to model tool calls. They do not prevent upstream's internal first-turn memory search and injection when `--memory` is enabled, so memory snippets can enter the xAI request even though direct model reads of `~/.grok/memory/**` are denied. Environment scrubbing and path-pattern denies do not isolate credentials or memory content. Model-facing MCP and meta-tool denies do not prove that native MCP servers, plugins, or hooks configured under `~/.grok` did not start during agent construction or had no side effects. The bridge variables gate only Grok's loose compatibility imports (skills, rules, agents, hooks, MCP servers, and sessions read from other tools' configuration), not native Grok configuration and not Claude Code plugins: unless `~/.grok/config.toml` sets `[claude_compat] imported = true`, Grok enables every plugin that `~/.claude/settings.json` enables, and since 1.0.52 an enabled and trusted plugin's hooks join the session at startup, so Claude Code plugin hooks and MCP servers can run inside a managed run before the first model call. Listing those plugins in `[plugins] disabled`, or setting `[claude_compat] imported = true`, keeps them out. Grok also passes the `env` block of `~/.claude/settings.json`, trusted project settings, and `.envrc` into tool subprocesses, so scrubbing the child environment does not keep those values out of write-mode shells; `[claude_compat] imported = true` stops that import as well. Startup can create directories and trust-boundary files inside the Grok home before confinement applies, so the sessions-only write grant describes the enforced phase. Narrower exposure requires a dedicated sandbox profile, an isolated Grok home, an authentication broker, or upstream path-level authorization. Under the runtime-socket deny policy (introduced between 1.0.4 and 1.0.13, still present at 1.0.41), strict refuses to start while `/var/run/docker.sock` is a symlink, which is the resting state of macOS Docker engines such as OrbStack; the companion preflights this and fails the dispatch as `sandbox` in seconds, and the repair is host side (for OrbStack `orb config set setup.use_admin false` and removing the link, for Docker Desktop disabling the default socket), never a sandbox downgrade. OrbStack's non-admin socket under `~/.orbstack/run` is outside upstream's deny list and the macOS strict profile allows network-outbound, so a strict write shell can drive that engine (verified at 1.0.41); the companion refuses write runs as `sandbox` while the socket exists, consult and review are unaffected, and the repair is quitting the engine or routing the package to Codex, whose sandbox blocks the connect. diff --git a/docs/grok-contract.md b/docs/grok-contract.md index 95eaba8..ffee113 100644 --- a/docs/grok-contract.md +++ b/docs/grok-contract.md @@ -48,7 +48,7 @@ Job statuses, liveness, rendered outcome footers, failure kind definitions, back - The companion's own tested interval is `[1.0.14, 1.0.42)`: below it managed runs fail closed as `setup`, at or past it `/grok:setup` reports the build as newer than the tested interval and a verification pass is advised; job records carry `grokVersion` so a breaker advisory can name the build that failed. - Strict's upstream `system_read` surface is broader than the workspace and Grok home. Linux includes `/var` and `/tmp`; macOS includes `/private` and additionally exposes the entire `~/Library`, so shared temporary content and substantial user application configuration and caches may be readable. Its write roots are the workspace and `~/.grok/sessions` (since 1.0.14 the rest of the Grok home is readable but not writable), always include `/tmp` and `/var/tmp`, and on macOS also include `/private/tmp`, `/private/var/tmp`, and all of `/private/var/folders`; the companion adds its unique run `TMPDIR` and verifies that it appears in the applied profile. The private directory is the adapter's preferred temporary path, not the only writable temporary surface. On Linux, a strict profile with deny paths re-executes under bubblewrap and refuses to start when `bwrap` is missing or unusable; macOS has no equivalent hard enforcement and relies on Seatbelt application plus best-effort tool denies. User toolchains outside strict's readable paths may be unavailable in write runs. Strict requests child process network restriction. Linux enforces it through seccomp, while the current macOS network blocker is a no-op, so write shell commands can still reach the network there. `--web` controls only Grok's built in web tools and may transmit inspected content to xAI services; it is not a shell network gate. The companion pins `GROK_WEB_FETCH=1` with `--web` and `GROK_WEB_FETCH=0` without it, and scrubs `GROK_WEB_FETCH_PROXY` and `GROK_WEB_FETCH_ALLOW_LOCAL`. The companion never retries under workspace or treats a missing host tool as permission to weaken the profile. Grok's memory-tracing diagnostics and models-cache refresh resolve the real OS home independently of a `$HOME` override on macOS (introduced in 0.2.112, verified through 1.0.30 on 2026-09-14; observed 2026-07-29), so redirecting `HOME` does not fully relocate `~/.grok` side effects; companion behavior is unaffected because it never overrides `HOME`. -- The Grok home is an upstream trust boundary, not private adapter state. The sandbox permits consult's `read_file` tool to reach files such as `~/.grok/auth.json`, configuration, and sessions, while write's terminal can receive the retained xAI authentication environment. Since 1.0.14 strict writes only `~/.grok/sessions` inside the home, so those files are readable but not writable from a managed run. The companion layers best-effort Read denies over `auth.json`, `mcp_credentials.json`, `config.toml`, `sessions`, `memory`, `logs`, and `debug` under the Grok home, using both absolute paths and standard `**/.grok` patterns. Raw path variants, symbolic links, and shell or indirect scripts can bypass those tool-level patterns, so they are exposure reduction rather than credential isolation. The Read denies apply to model tools and do not block upstream's internal first-turn memory injection when `--memory` is enabled. Scrubbing unrelated environment secrets does not isolate filesystem credentials, opted-in memory content, or the authentication Grok itself needs. Grok's builder specially preserves MCP meta-tools; the explicit `search_tool`, `use_tool`, and MCP denies block model invocation but cannot prove that native MCP servers, plugins, or hooks configured under `~/.grok` did not start during agent construction or had no side effects. The bridge variables gate only the loose compatibility imports, not native Grok configuration and not Claude Code plugins: unless `~/.grok/config.toml` sets `[claude_compat] imported = true`, Grok enables every plugin that `~/.claude/settings.json` enables, and since 1.0.52 an enabled and trusted plugin's hooks join the session at startup alongside its MCP servers, so they can run inside a managed run before the first model call. `[plugins] disabled` or `[claude_compat] imported = true` in `~/.grok/config.toml` keeps them out; the companion does not rewrite that file. Grok also passes the `env` block of `~/.claude/settings.json`, trusted project settings, and `.envrc` into tool subprocesses, so child environment scrubbing does not keep those values out of write-mode shells, and removing the web fetch proxy and local variables does not remove an allowance configured in TOML, which takes precedence over the environment. Startup can create directories and trust-boundary files inside the Grok home before confinement applies, so the sessions-only write grant describes the enforced phase. Narrower exposure requires a dedicated sandbox profile, an isolated Grok home, an authentication broker, or upstream path-level authorization; the current companion does not claim a hard path boundary. Grok offers configuration-layer narrowing primitives the companion does not currently inject per run (introduced in 0.2.112, verified through 1.0.30 on 2026-09-14): `[shell_environment_policy]` for the environment reaching shell tools, `[toolset.web_fetch] allowed_domains`, and hosted-tool `tool_overrides` domain and date fences. +- The Grok home is an upstream trust boundary, not private adapter state. The sandbox permits consult's `read_file` tool to reach files such as `~/.grok/auth.json`, configuration, and sessions, while write's terminal can receive the retained xAI authentication environment. Since 1.0.14 strict writes only `~/.grok/sessions` inside the home, so those files are readable but not writable from a managed run. The companion layers best-effort Read denies over `auth.json`, `mcp_credentials.json`, `config.toml`, `sessions`, `memory`, `logs`, and `debug` under the Grok home, using both absolute paths and standard `**/.grok` patterns. Raw path variants, symbolic links, and shell or indirect scripts can bypass those tool-level patterns, so they are exposure reduction rather than credential isolation. The Read denies apply to model tools and do not block upstream's internal first-turn memory injection when `--memory` is enabled. Scrubbing unrelated environment secrets does not isolate filesystem credentials, opted-in memory content, or the authentication Grok itself needs. Grok's builder specially preserves MCP meta-tools; the explicit `search_tool`, `use_tool`, and MCP denies block model invocation but cannot prove that native MCP servers, plugins, or hooks configured under `~/.grok` did not start during agent construction or had no side effects. The bridge variables gate only the loose compatibility imports, not native Grok configuration and not Claude Code plugins: unless `~/.grok/config.toml` sets `[claude_compat] imported = true`, Grok enables every plugin that `~/.claude/settings.json` enables, and since 1.0.52 an enabled and trusted plugin's hooks join the session at startup alongside its MCP servers, so they can run inside a managed run before the first model call. `[plugins] disabled` or `[claude_compat] imported = true` in `~/.grok/config.toml` keeps them out; the companion does not rewrite that file. Grok also passes the `env` block of `~/.claude/settings.json`, trusted project settings, and `.envrc` into tool subprocesses, so child environment scrubbing does not keep those values out of write-mode shells (`[claude_compat] imported = true` stops that import as well), and removing the web fetch proxy and local variables does not remove an allowance configured in TOML, which takes precedence over the environment. Startup can create directories and trust-boundary files inside the Grok home before confinement applies, so the sessions-only write grant describes the enforced phase. Narrower exposure requires a dedicated sandbox profile, an isolated Grok home, an authentication broker, or upstream path-level authorization; the current companion does not claim a hard path boundary. Grok offers configuration-layer narrowing primitives the companion does not currently inject per run (introduced in 0.2.112, verified through 1.0.30 on 2026-09-14): `[shell_environment_policy]` for the environment reaching shell tools, `[toolset.web_fetch] allowed_domains`, and hosted-tool `tool_overrides` domain and date fences. - Before every managed launch, the companion creates a unique private `TMPDIR`, snapshots Grok's shared `~/.grok/sessions/sandbox-events.jsonl` log (the home-root location that 1.0.13 and earlier wrote is unsupported), starts the child, and withholds the prompt until a new `ProfileApplied` event containing that private path also proves the expected canonical workspace, `strict` profile, enforced state, and requested `restrict_network: true` configuration. At 1.0.0, `ProfileApplied` also carries `platform` (`linux` with `landlock`, or `macos` with `seatbelt`), and the log can add `FsViolation`, `NetViolation`, `BypassGranted`, and `BypassDenied` event types. Those fields and events are additive, so the companion matcher is unaffected. There is no dedicated tmpdir field: temporary roots ride inside `read_write_paths`, which is exactly what the companion matches. Upstream events have no run id or pid, so the unique `TMPDIR` is the correlation key for positive evidence; unrelated `ProfileApplied` and `ApplyFailed` records are not attributed to the current run, and failure records remain auxiliary diagnostics. The matching event proves that the profile and configuration were applied and marked enforced, not that the platform blocked every network path. Upstream appends to `~/.grok/sessions/sandbox-events.jsonl` with no rotation or size cap, so the file grows unbounded; the companion's rotation and truncation detection therefore signals external interference, not normal upstream behavior. A matching warning on the owned stderr stream, a handshake timeout, missing or malformed matching evidence, shared-log disappearance or rotation, or any matching field mismatch terminates the owned process and fails closed as `sandbox`. Stderr can fail the owned handshake but cannot prove successful enforcement. - Upstream wires only `bypassPermissions` at spawn; the remaining `--permission-mode` values are forward-compatibility placeholders. The companion still passes `default` in headless consult runs, where a call that would prompt is cancelled, but consult isolation rests on the hard `--tools` filter and deny rules rather than permission mode. - `GROK_CONSULT_ALLOW` is retained only for read, grep, web search, and web fetch permission patterns. Shell, edit, MCP, and Agent rules are discarded, and the variable cannot add a tool omitted by the hard `--tools` filter.