From aab903aab9649e980b07cbf9f0bdb0a0a70dc4db Mon Sep 17 00:00:00 2001 From: Martin Patino Date: Mon, 5 Oct 2026 15:20:55 -0700 Subject: [PATCH 1/5] feat(runner): add OpenRouter as a gateway for any model OpenRouter joins DeepSeek and MiniMax as a gateway provider: the stock claude binary runs against https://openrouter.ai/api with the operator's OPENROUTER_API_KEY, an isolated config dir, and the empty ANTHROPIC_API_KEY that OpenRouter's Claude Code guide requires. Only OpenRouter gets the empty key, so DeepSeek and MiniMax env is unchanged. There is no model allowlist. The flex matrix carries one open openrouter row; each distinct OpenRouter model ID is its own family, and setup's live probe on the named model is the gate. The OpenRouter probe reads its marker from a file, so a pass proves a tool call. The runner refuses an ID without a namespace and OpenRouter's own openrouter/* routers, which pick the model server-side. OpenRouter reports must match the requested ID exactly apart from case, since a prefix rule would accept a sibling model. Panel diversity counts the lab behind the model: an OpenRouter lane counts as its ID's namespace. Refs #7 --- .../poteto-mode/references/codex-tools.md | 2 +- .../references/provider-dispatch.md | 23 +++--- .../poteto-mode/scripts/runner/cli.test.ts | 16 ++++- .../scripts/runner/commands.test.ts | 3 +- .../scripts/runner/flex-providers.test.ts | 49 +++++++++++++ .../scripts/runner/flex-providers.ts | 29 ++++++++ .../scripts/runner/model-matrix.test.ts | 16 ++++- .../scripts/runner/parse-output.test.ts | 10 +++ .../scripts/runner/parse-output.ts | 5 ++ .../poteto-mode/scripts/runner/run.test.ts | 72 ++++++++++++++++++- .../skills/poteto-mode/scripts/runner/run.ts | 5 ++ .../poteto-mode/scripts/runner/types.ts | 2 +- plugins/pstack/skills/setup-pstack/SKILL.md | 17 +++-- 13 files changed, 224 insertions(+), 25 deletions(-) diff --git a/plugins/pstack/skills/poteto-mode/references/codex-tools.md b/plugins/pstack/skills/poteto-mode/references/codex-tools.md index 68e49981..6260c0ce 100644 --- a/plugins/pstack/skills/poteto-mode/references/codex-tools.md +++ b/plugins/pstack/skills/poteto-mode/references/codex-tools.md @@ -42,7 +42,7 @@ poteto-mode's Subagents section sets Claude-specific defaults (`subagent_type: " ## Models and providers -Do not replace every configured entry with a Codex model. `/setup-pstack` writes portable descriptors such as `claude:fable@max`, `codex:gpt-5.6-sol@max`, and `grok:grok-4.7@xhigh`. In a Codex parent, only `codex:*` is native. Route Claude and Grok descriptors through the external launcher exactly as `provider-dispatch.md` specifies. The current default panel runs four lanes across three providers (Fable and Opus are both Claude) and contains no older GPT or Claude substitute. pstack-flex gateway descriptors (`deepseek:*`, `minimax:*`) also always route through the external launcher in a Codex parent; they are never `spawn_agent` lanes. +Do not replace every configured entry with a Codex model. `/setup-pstack` writes portable descriptors such as `claude:fable@max`, `codex:gpt-5.6-sol@max`, and `grok:grok-4.7@xhigh`. In a Codex parent, only `codex:*` is native. Route Claude and Grok descriptors through the external launcher exactly as `provider-dispatch.md` specifies. The current default panel runs four lanes across three providers (Fable and Opus are both Claude) and contains no older GPT or Claude substitute. pstack-flex gateway descriptors (`deepseek:*`, `minimax:*`, `openrouter:*`) also always route through the external launcher in a Codex parent; they are never `spawn_agent` lanes. ## Claude built-in skills pstack references diff --git a/plugins/pstack/skills/poteto-mode/references/provider-dispatch.md b/plugins/pstack/skills/poteto-mode/references/provider-dispatch.md index af81c2ba..8dc9a4b0 100644 --- a/plugins/pstack/skills/poteto-mode/references/provider-dispatch.md +++ b/plugins/pstack/skills/poteto-mode/references/provider-dispatch.md @@ -42,16 +42,19 @@ pstack-flex addition. These lanes are additive. A flex lane runs the stock `clau | deepseek-pro | deepseek | deepseek-v4-pro | high | low medium high xhigh max | DEEPSEEK_API_KEY | https://api.deepseek.com/anthropic | | minimax | minimax | MiniMax-M3 | high | low medium high xhigh max | MINIMAX_API_KEY | https://api.minimax.io/anthropic | | minimax-preview | minimax | MiniMax-M3.1-Flash-Preview | high | low medium high xhigh max | MINIMAX_API_KEY | https://api.minimax.io/anthropic | +| openrouter | openrouter | | high | low medium high xhigh max | OPENROUTER_API_KEY | https://openrouter.ai/api | A family identifies one `(provider, model)` pair, not an entire provider. The existing `deepseek` and `minimax` family names and descriptors remain valid. `deepseek-pro` and `minimax-preview` are additional choices with independent requested efforts. Multiple models from one provider still count as one provider for panel diversity. +The `openrouter` row is open. Its Model cell stands for any model ID in OpenRouter's catalog, written with its namespace, such as `openrouter:moonshotai/kimi-k3@high`. Split a descriptor at the first `:` and the last `@`: an OpenRouter model ID carries a `/` and can carry its own `:` or `~`, as in `openrouter:z-ai/glm-5.3:free@high`. There is no allowlist; setup's live probe on the chosen model is the gate. Each distinct OpenRouter model ID is its own family with its own requested effort and probe, and the row supplies the default and selectable efforts. The launcher refuses an ID without a namespace, and it refuses OpenRouter's own `openrouter/*` routers (`openrouter/auto`, `openrouter/free`) because they pick the model server-side; every model a router could pick is reachable by its own ID. OpenRouter maps the requested effort onto each model's reasoning controls ([reasoning tokens](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens), checked 2026-10-05); a receipt proves the request, not the applied depth. The runner sets `ANTHROPIC_API_KEY` to an empty string for this provider, as [OpenRouter's Claude Code guide](https://openrouter.ai/docs/guides/guides/claude-code-integration) requires. Data-collection and zero-data-retention routing, and the key's credit limit, are OpenRouter account settings, not pstack configuration. + MiniMax preview requires Token Plan access; set `MINIMAX_API_KEY` to the eligible subscription key. A pay-as-you-go key is not proof of preview access. The preview always thinks and supports `low` through `max`; do not disable thinking. M3 thinking is off by default at the API and requires adaptive thinking to enable it; its effort flag does not imply preview-style depth control. Selectable efforts are runner requests, not a claim that every provider applies five distinct reasoning levels. Verify CLI forwarding and model access with live probes. Sources: [MiniMax models](https://platform.minimax.io/docs/guides/models-intro), [MiniMax thinking controls](https://platform.minimax.io/docs/api-reference/text-anthropic-api), [DeepSeek Anthropic compatibility](https://api-docs.deepseek.com/guides/anthropic_api) (checked 2026-09-27). -Flex lanes have no Claude-native agent stem and always take the external runner in both parents. The base URL is a documented default; override it with `DEEPSEEK_BASE_URL` or `MINIMAX_BASE_URL`, and confirm it against the provider's current Claude Code guide during setup's live probe. The config dir defaults to `~/.pstack-flex/` (override: `PSTACK_FLEX__CONFIG_DIR`). Secrets stay in the environment: nothing in the sheet, the receipts, or this repository carries a key. +Flex lanes have no Claude-native agent stem and always take the external runner in both parents. The base URL is a documented default; override it with `DEEPSEEK_BASE_URL`, `MINIMAX_BASE_URL`, or `OPENROUTER_BASE_URL` (OpenRouter's must end in `/api`, not `/api/v1`), and confirm it against the provider's current Claude Code guide during setup's live probe. The config dir defaults to `~/.pstack-flex/` (override: `PSTACK_FLEX__CONFIG_DIR`). Secrets stay in the environment: nothing in the sheet, the receipts, or this repository carries a key. -Gateway receipt semantics differ from stock claude lanes in two documented ways. `costUsd` is always `null`: the claude CLI prices `total_cost_usd` at Anthropic rates, which would be fiction for third-party traffic; real prices live in [LANES.md](../../../../../docs/LANES.md), and token usage in the receipt stays accurate. Model verification accepts a case-insensitive matching provider report. A mismatched report fails the lane. When the endpoint reports no model, the receipt uses `modelEvidence: "pinned-argv"` and `modelVerified: false`. +Gateway receipt semantics differ from stock claude lanes in two documented ways. `costUsd` is always `null`: the claude CLI prices `total_cost_usd` at Anthropic rates, which would be fiction for third-party traffic; real prices live in [LANES.md](../../../../../docs/LANES.md), and token usage in the receipt stays accurate. Model verification accepts a case-insensitive matching provider report. An OpenRouter report must otherwise match exactly: any catalog model can be requested, so a prefix rule would accept a sibling such as `z-ai/glm-5.3-air` for `z-ai/glm-5.3`. A mismatched report fails the lane. When the endpoint reports no model, the receipt uses `modelEvidence: "pinned-argv"` and `modelVerified: false`. -Panel diversity rule (pstack-flex): `arena runners` and `interrogate reviewers` must span at least two distinct providers. DeepSeek plus MiniMax satisfies it. A single-provider panel is written only after the operator explicitly confirms the reduced diversity during setup, and the setup report records that confirmation. The adversarial signal comes from model diversity, so treat the override as an exception, not a configuration style. +Panel diversity rule (pstack-flex): `arena runners` and `interrogate reviewers` must span at least two distinct providers. DeepSeek plus MiniMax satisfies it. For this rule a lane's provider is the lab that made the model, not the route that reaches it. An `openrouter` lane counts as its model ID's namespace, and the namespaces `anthropic`, `openai`, `x-ai`, `deepseek`, and `minimax` count as the `claude`, `codex`, `grok`, `deepseek`, and `minimax` providers. So `openrouter:deepseek/deepseek-v4-pro` plus `deepseek:deepseek-flash` is one provider, and `openrouter:google/gemini-3.8-flash` plus `openrouter:z-ai/glm-5.3` is two. A single-provider panel is written only after the operator explicitly confirms the reduced diversity during setup, and the setup report records that confirmation. The adversarial signal comes from model diversity, so treat the override as an exception, not a configuration style. ## Read-time normalization @@ -65,10 +68,10 @@ This read-time rule makes an older installed sheet use the latest family revisio The top-level harness resolves the route once. A child receives an assigned provider, model, effort, access mode, prompt, working directory, and output path. A child never detects the harness, chooses a provider, or launches another model. Environment markers may corroborate the top-level harness before fan-out, but nested processes inherit parent markers and must not use them for routing. -| Parent | `claude:*` | `codex:*` | `grok:*` | `deepseek:*` | `minimax:*` | -|---|---|---|---|---|---| -| Claude Code | native `Agent` | external runner | external runner | external runner | external runner | -| Codex | external runner | native `spawn_agent` | external runner | external runner | external runner | +| Parent | `claude:*` | `codex:*` | `grok:*` | `deepseek:*` | `minimax:*` | `openrouter:*` | +|---|---|---|---|---|---|---| +| Claude Code | native `Agent` | external runner | external runner | external runner | external runner | external runner | +| Codex | external runner | native `spawn_agent` | external runner | external runner | external runner | external runner | Flex gateway descriptors are never native, even under a Claude Code parent: the gateway lane must run in its own process with injected endpoint, token, and isolated config dir, which the parent's native `Agent` primitive cannot provide. @@ -90,7 +93,7 @@ The launcher lives at `skills/poteto-mode/scripts/runner/pstack-runner` under th ```text pstack-runner \ --parent \ - --provider \ + --provider \ --model \ --effort \ --mode \ @@ -105,7 +108,7 @@ Pass arguments as an argv array or quote every path. Never interpolate prompt te pstack-flex: add `--label ` to name the lane in the agent monitor, using the role it fills, such as `--label "arena cross-judge"` or `--label "interrogate reviewer 2"`. The label is display text only; the launcher never routes on it. While the lane journal is on (starting [psf-monitor](https://github.com/thisguymartin/psf-monitor) turns it on), the launcher also records the lane's start, its output as it streams, and a copy of its receipt under `~/.pstack-flex/lanes/`, so psf-monitor can show the lane while it runs. A journal failure never changes the lane's receipt, exit status, or output. -Gateway lanes (`deepseek`, `minimax`) run three checks before the model executes, all fail-closed. First, in-process: the lane's API key variable must be set, and the lane's isolated `CLAUDE_CONFIG_DIR` must be free of OAuth credentials — a `.credentials.json` carrying a claude.ai login, or one that cannot be parsed, refuses the lane with an `unauthenticated` receipt before any subprocess runs, so a claude.ai credential can never be sent to a third-party endpoint. Second, the spawned preflight is `claude --version`, which proves the binary executes; `claude auth status` is deliberately not used because its behavior under token auth is undocumented. Third, the one-shot invocation is the real authentication and model test; an endpoint authentication error classifies as `unauthenticated` like any other lane. +Gateway lanes (`deepseek`, `minimax`, `openrouter`) run three checks before the model executes, all fail-closed. Before any of them, the launcher refuses an `openrouter` model ID without a namespace or from the `openrouter/*` routers as a usage error, so nothing is reserved or spawned. First, in-process: the lane's API key variable must be set, and the lane's isolated `CLAUDE_CONFIG_DIR` must be free of OAuth credentials — a `.credentials.json` carrying a claude.ai login, or one that cannot be parsed, refuses the lane with an `unauthenticated` receipt before any subprocess runs, so a claude.ai credential can never be sent to a third-party endpoint. Second, the spawned preflight is `claude --version`, which proves the binary executes; `claude auth status` is deliberately not used because its behavior under token auth is undocumented. Third, the one-shot invocation is the real authentication and model test; an endpoint authentication error classifies as `unauthenticated` like any other lane. Grok authentication preflight has one bounded retry. If the first `grok models` result would be classified as unauthenticated, the runner waits five seconds and tries the same preflight once more. A second failure is terminal. The delay and second attempt share the runner's absolute deadline and cancellation latch, and the receipt keeps evidence from both attempts. Model execution is never retried. @@ -130,7 +133,7 @@ Success requires all of these: 1. Exit status `0`. 2. Receipt status `complete`. -3. Either `modelVerified: true` with `modelEvidence: "provider-report"`, or a Codex receipt with `reportedModel: null`, `modelVerified: false`, and `modelEvidence: "pinned-argv"`, or a gateway (`deepseek`/`minimax`) receipt with `modelVerified: false` and `modelEvidence: "pinned-argv"` when the endpoint does not echo the requested slug. For Claude's `fable` and `opus` aliases, the concrete provider report must belong to the requested family. Codex 0.149.0 accepts the exact `--model` argument but does not report the served model in its JSONL stream. Gateway reports match case-insensitively because third-party endpoints are inconsistent about slug casing. +3. Either `modelVerified: true` with `modelEvidence: "provider-report"`, or a Codex receipt with `reportedModel: null`, `modelVerified: false`, and `modelEvidence: "pinned-argv"`, or a gateway (`deepseek`/`minimax`/`openrouter`) receipt with `modelVerified: false` and `modelEvidence: "pinned-argv"` when the endpoint does not echo the requested slug. For Claude's `fable` and `opus` aliases, the concrete provider report must belong to the requested family. Codex 0.149.0 accepts the exact `--model` argument but does not report the served model in its JSONL stream. Gateway reports match case-insensitively because third-party endpoints are inconsistent about slug casing; an OpenRouter report must otherwise match the requested ID exactly. 4. A non-empty output file. The receipt also carries elapsed time, token usage when the CLI exposes it, and cost when available. Keep it with the arena or review artifacts so parent-harness comparisons are evidence-based. diff --git a/plugins/pstack/skills/poteto-mode/scripts/runner/cli.test.ts b/plugins/pstack/skills/poteto-mode/scripts/runner/cli.test.ts index ceea65cd..53ec3ee9 100644 --- a/plugins/pstack/skills/poteto-mode/scripts/runner/cli.test.ts +++ b/plugins/pstack/skills/poteto-mode/scripts/runner/cli.test.ts @@ -55,6 +55,20 @@ describe("runner CLI parsing", () => { expect(parsed?.model).toBe("MiniMax-M3"); }); + it("passes a namespaced OpenRouter model through unchanged", () => { + const parsed = parseArgs( + argv().map((value, index, all) => + all[index - 1] === "--provider" + ? "openrouter" + : all[index - 1] === "--model" + ? "moonshotai/kimi-k3" + : value + ) + ); + expect(parsed?.provider).toBe("openrouter"); + expect(parsed?.model).toBe("moonshotai/kimi-k3"); + }); + it("takes an optional display label and nothing else from it", () => { expect(parseArgs(argv())?.label).toBeUndefined(); expect(parseArgs(argv(["--label", " arena cross-judge "]))?.label).toBe("arena cross-judge"); @@ -69,6 +83,6 @@ describe("runner CLI parsing", () => { all[index - 1] === "--provider" ? "gemini" : value ) ) - ).toThrow("deepseek, minimax"); + ).toThrow("deepseek, minimax, openrouter"); }); }); diff --git a/plugins/pstack/skills/poteto-mode/scripts/runner/commands.test.ts b/plugins/pstack/skills/poteto-mode/scripts/runner/commands.test.ts index 827a1b66..e6b15e72 100644 --- a/plugins/pstack/skills/poteto-mode/scripts/runner/commands.test.ts +++ b/plugins/pstack/skills/poteto-mode/scripts/runner/commands.test.ts @@ -150,6 +150,7 @@ describe("invocationCommand", () => { for (const [provider, model] of [ ["deepseek", "deepseek-flash"], ["minimax", "MiniMax-M3"], + ["openrouter", "z-ai/glm-5.3"], ] as const) { const gateway = invocationCommand(options({ provider, model })); const claude = invocationCommand( @@ -162,7 +163,7 @@ describe("invocationCommand", () => { }); it("preflights gateway lanes with a version probe, not an auth check", () => { - for (const provider of ["deepseek", "minimax"] as const) { + for (const provider of ["deepseek", "minimax", "openrouter"] as const) { const spec = preflightCommand(provider); expect(spec.command).toBe("claude"); expect(spec.args).toEqual(["--version"]); diff --git a/plugins/pstack/skills/poteto-mode/scripts/runner/flex-providers.test.ts b/plugins/pstack/skills/poteto-mode/scripts/runner/flex-providers.test.ts index 755b70d3..362665a4 100644 --- a/plugins/pstack/skills/poteto-mode/scripts/runner/flex-providers.test.ts +++ b/plugins/pstack/skills/poteto-mode/scripts/runner/flex-providers.test.ts @@ -8,6 +8,7 @@ import { gatewayConfigDir, gatewayEnvironment, gatewayGuard, + openRouterModelRefusal, } from "./flex-providers.ts"; import { GATEWAY_PROVIDERS } from "./types.ts"; @@ -39,6 +40,9 @@ describe("gatewayConfigDir", () => { expect(gatewayConfigDir("minimax", {})).toBe( join(homedir(), ".pstack-flex", "minimax") ); + expect(gatewayConfigDir("openrouter", {})).toBe( + join(homedir(), ".pstack-flex", "openrouter") + ); }); it("honors the override variable and ignores blank overrides", () => { @@ -72,6 +76,26 @@ describe("gatewayEnvironment", () => { }); }); + it("injects OpenRouter's endpoint with an explicitly empty API key", () => { + const source = { + OPENROUTER_API_KEY: "sk-or-test", + PSTACK_FLEX_OPENROUTER_CONFIG_DIR: scratch, + }; + expect(gatewayEnvironment("openrouter", "z-ai/glm-5.3", source)).toEqual({ + ANTHROPIC_BASE_URL: "https://openrouter.ai/api", + ANTHROPIC_AUTH_TOKEN: "sk-or-test", + ANTHROPIC_API_KEY: "", + ANTHROPIC_MODEL: "z-ai/glm-5.3", + ANTHROPIC_DEFAULT_OPUS_MODEL: "z-ai/glm-5.3", + ANTHROPIC_DEFAULT_SONNET_MODEL: "z-ai/glm-5.3", + ANTHROPIC_DEFAULT_HAIKU_MODEL: "z-ai/glm-5.3", + CLAUDE_CODE_SUBAGENT_MODEL: "z-ai/glm-5.3", + CLAUDE_CODE_ATTRIBUTION_HEADER: "0", + CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC: "1", + CLAUDE_CONFIG_DIR: scratch, + }); + }); + it("omits the context cap when the provider has no default", () => { const env = gatewayEnvironment("minimax", "MiniMax-M3", { MINIMAX_API_KEY: "mm-test", @@ -120,6 +144,31 @@ describe("gatewayEnvironment", () => { }); }); +describe("openRouterModelRefusal", () => { + it("accepts any namespaced catalog ID", () => { + for (const model of [ + "z-ai/glm-5.3", + "moonshotai/kimi-k3", + "anthropic/claude-sonnet-5.5", + "qwen/qwen3.8-max-0902", + "~google/gemini-flash-latest", + "z-ai/glm-5.3:free", + ]) expect(openRouterModelRefusal(model)).toBeNull(); + }); + + it("refuses a bare slug", () => { + for (const model of ["glm-5.3", "/glm-5.3", "z-ai/"]) { + expect(openRouterModelRefusal(model)).toContain("must be a namespaced ID"); + } + }); + + it("refuses OpenRouter's own routers, which pick the model server-side", () => { + for (const model of ["openrouter/auto", "openrouter/free", "OpenRouter/auto-beta"]) { + expect(openRouterModelRefusal(model)).toContain("picks the model server-side"); + } + }); +}); + describe("gatewayGuard", () => { it("refuses when the API key variable is missing or blank", () => { expect(gatewayGuard("deepseek", {})?.message).toBe("DEEPSEEK_API_KEY is not set"); diff --git a/plugins/pstack/skills/poteto-mode/scripts/runner/flex-providers.ts b/plugins/pstack/skills/poteto-mode/scripts/runner/flex-providers.ts index 1a247860..7dccf69c 100644 --- a/plugins/pstack/skills/poteto-mode/scripts/runner/flex-providers.ts +++ b/plugins/pstack/skills/poteto-mode/scripts/runner/flex-providers.ts @@ -15,6 +15,9 @@ export interface GatewaySpec { readonly configDirOverrideVar: string; readonly maxContextTokensDefault: string | null; readonly maxContextTokensOverrideVar: string; + // OpenRouter's Claude Code guide requires ANTHROPIC_API_KEY to be set and + // empty, not merely unset. + readonly emptyApiKey: boolean; } export const GATEWAY_SPECS: Record = { @@ -25,6 +28,7 @@ export const GATEWAY_SPECS: Record = { configDirOverrideVar: "PSTACK_FLEX_DEEPSEEK_CONFIG_DIR", maxContextTokensDefault: "128000", maxContextTokensOverrideVar: "DEEPSEEK_MAX_CONTEXT_TOKENS", + emptyApiKey: false, }, minimax: { apiKeyVar: "MINIMAX_API_KEY", @@ -33,9 +37,33 @@ export const GATEWAY_SPECS: Record = { configDirOverrideVar: "PSTACK_FLEX_MINIMAX_CONFIG_DIR", maxContextTokensDefault: null, maxContextTokensOverrideVar: "MINIMAX_MAX_CONTEXT_TOKENS", + emptyApiKey: false, + }, + openrouter: { + apiKeyVar: "OPENROUTER_API_KEY", + baseUrlDefault: "https://openrouter.ai/api", + baseUrlOverrideVar: "OPENROUTER_BASE_URL", + configDirOverrideVar: "PSTACK_FLEX_OPENROUTER_CONFIG_DIR", + maxContextTokensDefault: null, + maxContextTokensOverrideVar: "OPENROUTER_MAX_CONTEXT_TOKENS", + emptyApiKey: true, }, }; +// OpenRouter serves any catalog model by its namespaced ID, such as +// `z-ai/glm-5.3`. Its own `openrouter/*` IDs (auto, free) pick the model +// server-side, which would hide which model ran. +export function openRouterModelRefusal(model: string): string | null { + const slash = model.indexOf("/"); + if (slash <= 0 || slash === model.length - 1) { + return `OpenRouter model ${model} must be a namespaced ID such as z-ai/glm-5.3`; + } + if (model.slice(0, slash).toLowerCase() === "openrouter") { + return `OpenRouter router ${model} picks the model server-side; name the model directly`; + } + return null; +} + // Provider selection and Claude configuration from the parent must not // override the gateway's endpoint, token, or isolated config directory. export const GATEWAY_INHERITED_CONFLICTS = [ @@ -84,6 +112,7 @@ export function gatewayEnvironment( }; const token = overridden(source, spec.apiKeyVar); if (token !== null) injected.ANTHROPIC_AUTH_TOKEN = token; + if (spec.emptyApiKey) injected.ANTHROPIC_API_KEY = ""; const maxContext = overridden(source, spec.maxContextTokensOverrideVar) ?? spec.maxContextTokensDefault; if (maxContext !== null) injected.CLAUDE_CODE_MAX_CONTEXT_TOKENS = maxContext; diff --git a/plugins/pstack/skills/poteto-mode/scripts/runner/model-matrix.test.ts b/plugins/pstack/skills/poteto-mode/scripts/runner/model-matrix.test.ts index 6e849198..01b6ae7b 100644 --- a/plugins/pstack/skills/poteto-mode/scripts/runner/model-matrix.test.ts +++ b/plugins/pstack/skills/poteto-mode/scripts/runner/model-matrix.test.ts @@ -67,6 +67,7 @@ const SETUP_SECTION_ORDER = [ "### 7. Confirm and commit", ] as const; +const OPEN_OPENROUTER_MODEL = ""; const FLEX_MATRIX_HEADER = [ "Family", "Provider", @@ -512,7 +513,13 @@ describe("model matrix", () => { const pair = `${provider}:${model}`; expect(pairs.has(pair)).toBe(false); pairs.add(pair); - expect(/^[A-Za-z0-9.-]+$/.test(model)).toBe(true); + // OpenRouter's row is open: its Model cell is a placeholder for any + // catalog ID, which setup probes one model at a time. + if (gateway === "openrouter") { + expect(model).toBe(OPEN_OPENROUTER_MODEL); + } else { + expect(/^[A-Za-z0-9.-]+$/.test(model)).toBe(true); + } const selectable = selectableRaw.split(/\s+/).map(asEffort); expect(selectable).toContain(asEffort(defaultEffortRaw)); expect(keyVar).toBe(GATEWAY_SPECS[gateway].apiKeyVar); @@ -525,13 +532,18 @@ describe("model matrix", () => { "deepseek:deepseek-v4-pro", "minimax:MiniMax-M3", "minimax:MiniMax-M3.1-Flash-Preview", + `openrouter:${OPEN_OPENROUTER_MODEL}`, ]) expect(pairs.has(pair)).toBe(true); + expect(dispatch).toContain("The `openrouter` row is open."); + expect(dispatch).toContain("There is no allowlist; setup's live probe on the chosen model is the gate."); + expect(dispatch).toContain("Each distinct OpenRouter model ID is its own family"); + expect(dispatch).toContain("a lane's provider is the lab that made the model, not the route that reaches it."); expect(setup).toContain("Never group efforts or deduplicate probes by provider alone."); expect(setup).toContain("Different models sharing a provider count as one provider"); // The stock quad and first-run sheet must not carry flex descriptors: // upstream's own checks parse descriptors with a lowercase-only, // three-provider grammar and must never see a flex lane. - expect(firstRunSheet(setup)).not.toMatch(/deepseek:|minimax:/i); + expect(firstRunSheet(setup)).not.toMatch(/deepseek:|minimax:|openrouter:/i); }); it("binds Claude-native dispatch to the matrix mapping", () => { diff --git a/plugins/pstack/skills/poteto-mode/scripts/runner/parse-output.test.ts b/plugins/pstack/skills/poteto-mode/scripts/runner/parse-output.test.ts index 270a5863..bd4feabc 100644 --- a/plugins/pstack/skills/poteto-mode/scripts/runner/parse-output.test.ts +++ b/plugins/pstack/skills/poteto-mode/scripts/runner/parse-output.test.ts @@ -142,6 +142,16 @@ describe("parseProviderOutput", () => { expect(reportedModelMatches("claude", "MiniMax-M3", "minimax-m3")).toBe(false); }); + it("matches OpenRouter models exactly, ignoring only case", () => { + expect(reportedModelMatches("openrouter", "z-ai/glm-5.3", "z-ai/glm-5.3")).toBe(true); + expect(reportedModelMatches("openrouter", "z-ai/glm-5.3", "Z-AI/GLM-5.3")).toBe(true); + expect(reportedModelMatches("openrouter", "z-ai/glm-5.3", "z-ai/glm-5.3-air")).toBe(false); + expect( + reportedModelMatches("openrouter", "google/gemini-3.8-flash", "google/gemini-3.8-flash-lite") + ).toBe(false); + expect(reportedModelMatches("openrouter", "z-ai/glm-5.3", "glm-5.3")).toBe(false); + }); + it("matches only concrete Claude revisions from the requested rolling family", () => { expect(reportedModelMatches("claude", "fable", "claude-fable-9-9")).toBe(true); expect(reportedModelMatches("claude", "opus", "claude-opus-9")).toBe(true); diff --git a/plugins/pstack/skills/poteto-mode/scripts/runner/parse-output.ts b/plugins/pstack/skills/poteto-mode/scripts/runner/parse-output.ts index 64d51b75..dec15c17 100644 --- a/plugins/pstack/skills/poteto-mode/scripts/runner/parse-output.ts +++ b/plugins/pstack/skills/poteto-mode/scripts/runner/parse-output.ts @@ -194,6 +194,11 @@ export function reportedModelMatches( if (provider === "claude" && isRollingClaudeAlias(requested)) { return concreteModelMatchesRollingAlias(requested, reported); } + if (provider === "openrouter") { + // Any catalog model can be requested, so a prefix match would accept a + // sibling such as z-ai/glm-5.3-air for z-ai/glm-5.3. + return reported.toLowerCase() === requested.toLowerCase(); + } if (isGatewayProvider(provider)) { // Third-party endpoints are inconsistent about slug casing // (e.g. MiniMax-M3 vs minimax-m3); compare case-insensitively. diff --git a/plugins/pstack/skills/poteto-mode/scripts/runner/run.test.ts b/plugins/pstack/skills/poteto-mode/scripts/runner/run.test.ts index 00da27a3..5b6d2cd7 100644 --- a/plugins/pstack/skills/poteto-mode/scripts/runner/run.test.ts +++ b/plugins/pstack/skills/poteto-mode/scripts/runner/run.test.ts @@ -1008,8 +1008,10 @@ describe("gateway lanes", () => { const GATEWAY_TEST_KEYS = [ "DEEPSEEK_API_KEY", "MINIMAX_API_KEY", + "OPENROUTER_API_KEY", "PSTACK_FLEX_DEEPSEEK_CONFIG_DIR", "PSTACK_FLEX_MINIMAX_CONFIG_DIR", + "PSTACK_FLEX_OPENROUTER_CONFIG_DIR", "ANTHROPIC_API_KEY", "ANTHROPIC_CUSTOM_HEADERS", "CLAUDE_CODE_USE_BEDROCK", @@ -1019,15 +1021,21 @@ describe("gateway lanes", () => { "FAKE_OMIT_MODEL_USAGE", ] as const; + const GATEWAY_MODELS = { + deepseek: "deepseek-flash", + minimax: "MiniMax-M3", + openrouter: "z-ai/glm-5.3", + } as const; + function gatewayOptions( - provider: "deepseek" | "minimax", + provider: keyof typeof GATEWAY_MODELS, suffix: string ): RunnerOptions { return { ...options(provider === "deepseek" ? "claude" : "codex", suffix), provider, parent: "claude", - model: provider === "deepseek" ? "deepseek-flash" : "MiniMax-M3", + model: GATEWAY_MODELS[provider], effort: "high", }; } @@ -1038,6 +1046,8 @@ describe("gateway lanes", () => { process.env.MINIMAX_API_KEY = "sk-minimax-test"; process.env.PSTACK_FLEX_DEEPSEEK_CONFIG_DIR = join(scratch, "flex-deepseek"); process.env.PSTACK_FLEX_MINIMAX_CONFIG_DIR = join(scratch, "flex-minimax"); + process.env.OPENROUTER_API_KEY = "sk-or-test"; + process.env.PSTACK_FLEX_OPENROUTER_CONFIG_DIR = join(scratch, "flex-openrouter"); }); afterEach(() => { @@ -1195,6 +1205,64 @@ describe("gateway lanes", () => { expect(written.modelEvidence).toBe("pinned-argv"); }); + it("injects OpenRouter's endpoint and replaces the parent's API key with an empty one", async () => { + process.env.ANTHROPIC_API_KEY = "parent-anthropic-secret"; + const dumpPath = join(scratch, "openrouter-env.json"); + process.env.FAKE_DUMP_ENV_PATH = dumpPath; + const input = gatewayOptions("openrouter", "openrouter-env"); + expect((await runLane(input)).exitCode).toBe(0); + const child = JSON.parse(readFileSync(dumpPath, "utf8")) as Record; + expect(child.ANTHROPIC_BASE_URL).toBe("https://openrouter.ai/api"); + expect(child.ANTHROPIC_AUTH_TOKEN).toBe("sk-or-test"); + expect(child.ANTHROPIC_API_KEY).toBe(""); + expect(child.ANTHROPIC_MODEL).toBe("z-ai/glm-5.3"); + expect(child.CLAUDE_CONFIG_DIR).toBe(join(scratch, "flex-openrouter")); + expect(JSON.stringify(child)).not.toContain("parent-anthropic-secret"); + }); + + for (const parent of ["claude", "codex"] as const) { + it(`pins a namespaced OpenRouter model through the ${parent} parent route`, async () => { + const model = "moonshotai/kimi-k3"; + const input: RunnerOptions = { + ...gatewayOptions("openrouter", "openrouter-pin"), parent, model, effort: "low", + }; + expect((await runLane(input)).exitCode).toBe(0); + const written = receipt(input.receiptPath); + expect(written).toMatchObject({ + status: "complete", parent, provider: "openrouter", model, effort: "low", + reportedModel: model, modelVerified: true, modelEvidence: "provider-report", + costUsd: null, + }); + expect(written.argv[written.argv.indexOf("--model") + 1]).toBe(model); + }); + } + + it("fails an OpenRouter lane that reports a sibling of the requested model", async () => { + process.env.FAKE_REPORT_MODEL = "z-ai/glm-5.3-air"; + const input = gatewayOptions("openrouter", "openrouter-sibling"); + expect((await runLane(input)).exitCode).toBe(65); + const written = receipt(input.receiptPath); + expect(written.status).toBe("malformed-output"); + expect(written.modelVerified).toBe(false); + expect(existsSync(input.outputPath)).toBe(false); + }); + + it("refuses OpenRouter's auto router before reserving any output", async () => { + const input = { ...gatewayOptions("openrouter", "openrouter-auto"), model: "openrouter/auto" }; + await expect(runLane(input)).rejects.toThrow("picks the model server-side"); + expect(existsSync(input.outputPath)).toBe(false); + expect(existsSync(input.receiptPath)).toBe(false); + }); + + it("refuses an OpenRouter lane without its key and names the variable", async () => { + delete process.env.OPENROUTER_API_KEY; + const input = gatewayOptions("openrouter", "openrouter-missing-key"); + expect((await runLane(input)).exitCode).toBe(77); + const written = receipt(input.receiptPath); + expect(written.status).toBe("unauthenticated"); + expect(written.error?.message).toBe("OPENROUTER_API_KEY is not set"); + }); + it("classifies an endpoint authentication error as unauthenticated", async () => { process.env.FAKE_AUTH_ERROR = "1"; const input = gatewayOptions("deepseek", "endpoint-401"); diff --git a/plugins/pstack/skills/poteto-mode/scripts/runner/run.ts b/plugins/pstack/skills/poteto-mode/scripts/runner/run.ts index 0aacd0eb..8ec7aeb8 100644 --- a/plugins/pstack/skills/poteto-mode/scripts/runner/run.ts +++ b/plugins/pstack/skills/poteto-mode/scripts/runner/run.ts @@ -15,6 +15,7 @@ import { GATEWAY_INHERITED_CONFLICTS, gatewayEnvironment, gatewayGuard, + openRouterModelRefusal, } from "./flex-providers.ts"; import { versionedClaudeAlias } from "./model-aliases.ts"; import { parseProviderOutput, reportedModelMatches } from "./parse-output.ts"; @@ -537,6 +538,10 @@ export function validateOptions(options: RunnerOptions): void { `Claude model ${options.model} is a version pin; normalize it to ${staleAlias} before invoking the runner` ); } + const routerRefusal = options.provider === "openrouter" + ? openRouterModelRefusal(options.model) + : null; + if (routerRefusal !== null) throw new UsageError(routerRefusal); if ( options.timeoutMs !== null && (!Number.isFinite(options.timeoutMs) || options.timeoutMs <= 0) diff --git a/plugins/pstack/skills/poteto-mode/scripts/runner/types.ts b/plugins/pstack/skills/poteto-mode/scripts/runner/types.ts index 214d540d..3cfe1d9a 100644 --- a/plugins/pstack/skills/poteto-mode/scripts/runner/types.ts +++ b/plugins/pstack/skills/poteto-mode/scripts/runner/types.ts @@ -2,7 +2,7 @@ export const PARENTS = ["claude", "codex"] as const; // pstack-flex: gateway providers run the stock `claude` binary against a // third-party Anthropic-compatible endpoint with injected environment. Adding // one here requires a matching row in flex-providers.ts GATEWAY_SPECS. -export const GATEWAY_PROVIDERS = ["deepseek", "minimax"] as const; +export const GATEWAY_PROVIDERS = ["deepseek", "minimax", "openrouter"] as const; export const PROVIDERS = ["claude", "codex", "grok", ...GATEWAY_PROVIDERS] as const; export const EFFORTS = ["low", "medium", "high", "xhigh", "max"] as const; export const ACCESS_MODES = ["read-only", "isolated-write"] as const; diff --git a/plugins/pstack/skills/setup-pstack/SKILL.md b/plugins/pstack/skills/setup-pstack/SKILL.md index f7ff08d4..a49cd896 100644 --- a/plugins/pstack/skills/setup-pstack/SKILL.md +++ b/plugins/pstack/skills/setup-pstack/SKILL.md @@ -1,6 +1,6 @@ --- name: setup-pstack -description: Configure pstack's provider-qualified models, per-family requested effort, and parent-owned routes per role. Verifies native and external Claude, Codex, Grok, DeepSeek, and MiniMax lanes before writing the override sheet. Use for /setup-pstack, "configure pstack models", or changing pstack's model choices. +description: Configure pstack's provider-qualified models, per-family requested effort, and parent-owned routes per role. Verifies native and external Claude, Codex, Grok, DeepSeek, MiniMax, and OpenRouter lanes before writing the override sheet. Use for /setup-pstack, "configure pstack models", or changing pstack's model choices. --- # Setup pstack @@ -33,17 +33,17 @@ Read the current parent-specific sheet when it exists. Before matrix validation, Treat the normalized values as current role-to-family assignments. Overlay those rows on the complete first-run role map in step 7. Materialize any missing documented role row from that map on the next successful write. A duplicate role row is inconsistent state; report it and resolve it before probing. A row whose role is not in the step 7 role map, such as `how critics`, is from a retired role. Drop it and list it at confirmation. A bare host-native slug from an older sheet is also invalid because it does not say which provider owns it. A versioned Claude model outside the two migration families remains inconsistent state. If the sheet is missing, use the complete first-run role map and the model matrix's Default effort cells. -Then ask whether to keep these role-to-family assignments or change named roles. Keeping them is the default. Apply only role changes the operator names; never offer a reset of a customized sheet to the first-run assignments. A changed role may use any stock or flex matrix family, `inherit-parent`, or `auto`. +Then ask whether to keep these role-to-family assignments or change named roles. Keeping them is the default. Apply only role changes the operator names; never offer a reset of a customized sheet to the first-run assignments. A changed role may use any stock or flex matrix family, any OpenRouter model ID the operator names (`openrouter:/`), `inherit-parent`, or `auto`. Offer every stock family, including Astra, GPT-6 Sol, and Luna, when changing `architect runners` or another configurable role. Read each model, proposed effort, and selectable efforts from its row. The Codex families are separate families even though they share the Codex provider; changing one family's effort does not change another's. GPT-6 Sol uses the `sol-6` family; the `sol` family keeps GPT-5.6 Sol for sheets that still assign it. ### 3. Parse per-family efforts -Read the model matrices, stock and flex. Every non-alias value must match `:@`. Map it to exactly one matrix family by `(provider, model)`, require its effort to appear in that row's Selectable efforts cell, and collect the effort. `inherit-parent` and `auto` rows carry no family effort. +Read the model matrices, stock and flex. Every non-alias value must match `:@`. Map it to exactly one matrix family by `(provider, model)`; an `openrouter` descriptor maps to the open `openrouter` row whatever its model ID. Require its effort to appear in that row's Selectable efforts cell, and collect the effort. `inherit-parent` and `auto` rows carry no family effort. An unmatched provider/model, out-of-domain effort, or duplicate role is inconsistent state. Stop, show the conflicting rows verbatim, and ask for an explicit matrix family or alias replacement. If one or more families have mixed efforts, show every conflicting family and role row, then ask for one normalized effort per family from its Selectable efforts cell. Do not invent a precedence rule. Do not probe or write while any inconsistency is unresolved. -A family is a single `(provider, model)` matrix row. DeepSeek Flash and Pro have independent efforts, as do MiniMax M3 and M3.1 Flash Preview. Never group efforts or deduplicate probes by provider alone. +A family is a single `(provider, model)` matrix row. DeepSeek Flash and Pro have independent efforts, as do MiniMax M3 and M3.1 Flash Preview. Each distinct OpenRouter model ID is its own family under the open row, with its own effort and probe. Never group efforts or deduplicate probes by provider alone. One distinct effort per family is the current value. A family with no non-alias occurrence is unassigned: do not ask for its effort, check its CLI, or probe it. A family that a step 2 role change newly assigns takes its matrix Default effort as the proposed value. @@ -66,10 +66,13 @@ Probe only the selected `provider:model@effort` pair of each assigned family. Ru | Opus | Opus matrix row + selected effort | native Agent `pstack-opus-` | Claude CLI | native one-turn probe or `claude auth status --json` plus one-turn probe | | DeepSeek Flash / Pro | Each assigned DeepSeek flex row + selected effort | external runner | external runner | `DEEPSEEK_API_KEY` present; isolated config dir free of OAuth credentials; one-turn probe confirms the endpoint | | MiniMax M3 / M3.1 Flash Preview | Each assigned MiniMax flex row + selected effort | external runner | external runner | `MINIMAX_API_KEY` present; isolated config dir free of OAuth credentials; one-turn probe confirms the endpoint | +| OpenRouter (any model ID) | Each assigned OpenRouter model ID + selected effort | external runner | external runner | `OPENROUTER_API_KEY` present; isolated config dir free of OAuth credentials; one-turn probe that reads its marker from a file | For MiniMax M3.1 Flash Preview, disclose the Token Plan requirement before probing. Use the eligible subscription key through `MINIMAX_API_KEY`; do not assume a working M3 key grants preview access. A failed preview probe must not silently select M3. Keep preview thinking enabled and verify requested effort forwarding; distinguish request evidence from hidden applied reasoning depth. -Use a tiny read-only probe that returns a unique marker. A login-status command alone proves credentials, not that the requested model and effort flags run. Record native and external results separately. Never call the external launcher for the parent's own provider. On a Claude parent, the Fable and Opus probes are one-turn runs of the mapped `pstack--` agent. On a Codex parent, each assigned Codex family gets a native `spawn_agent` probe with its matrix model and selected `reasoning_effort`. Every other pair, flex families always included, uses the external runner with the selected effort flag. A flex probe doubles as the base-URL confirmation: it proves the documented default (or the operator's override) actually serves the lane's model. +Before the first OpenRouter probe, tell the operator three things: OpenRouter forwards prompts to whichever host serves the model, data-collection and zero-data-retention routing plus the key's credit limit are set on OpenRouter's dashboard, and each probe spends a little credit. Probe exactly the model ID the operator named. A failed OpenRouter probe does not offer another catalog model; report OpenRouter's error and ask for a replacement ID or a role reassignment. + +Use a tiny read-only probe that returns a unique marker. For an OpenRouter model, write the marker to a file in a scratch directory and leave it out of the prompt, so a passing probe proves the model made a tool call; Claude Code lanes cannot work without one. A login-status command alone proves credentials, not that the requested model and effort flags run. Record native and external results separately. Never call the external launcher for the parent's own provider. On a Claude parent, the Fable and Opus probes are one-turn runs of the mapped `pstack--` agent. On a Codex parent, each assigned Codex family gets a native `spawn_agent` probe with its matrix model and selected `reasoning_effort`. Every other pair, flex families always included, uses the external runner with the selected effort flag. A flex probe doubles as the base-URL confirmation: it proves the documented default (or the operator's override) actually serves the lane's model. Receipts and native transcripts prove the requested effort and the route. They do not prove a provider's hidden applied reasoning depth. There is no implicit timeout, weaker-model fallback, same-provider external fallback, or second mutable configuration source. @@ -82,11 +85,11 @@ Build the new sheet in memory. Do not write it yet. Require every documented role to remain present and non-empty, `architect runners` to keep at least two entries, and the final role map to contain at least one assigned matrix family. There is no requirement to assign every matrix family. -Different models sharing a provider count as one provider, even when their efforts differ. +Different models sharing a provider count as one provider, even when their efforts differ. An OpenRouter lane counts as its model ID's lab, as `provider-dispatch.md` defines: the namespaces `anthropic`, `openai`, `x-ai`, `deepseek`, and `minimax` match the direct providers, and any other namespace is a provider of its own. Validate panel diversity: `arena runners` and `interrogate reviewers` must span at least two distinct providers. A single-provider panel is written only after the operator explicitly confirms the reduced diversity; record that confirmation in the setup report. -Rewrite every matrix-family descriptor to `provider:model@`. Leave `inherit-parent` and `auto` unchanged. An effort-only rerun cannot change a role's family. Changing Grok's effort updates every Grok occurrence and does not move a Sol role onto Grok. Refuse an unqualified slug, an unavailable route, a model outside the stock and flex matrix families, or a provider/model mismatch. +Rewrite every matrix-family descriptor to `provider:model@`. Leave `inherit-parent` and `auto` unchanged. An effort-only rerun cannot change a role's family. Changing Grok's effort updates every Grok occurrence and does not move a Sol role onto Grok. Refuse an unqualified slug, an unavailable route, a model outside the stock and flex matrix families, an OpenRouter ID without a namespace or from the `openrouter/*` routers, or a provider/model mismatch. ### 7. Confirm and commit From 1922270dd0e07eef2e856d8db8052899ecadca9b Mon Sep 17 00:00:00 2001 From: Martin Patino Date: Mon, 5 Oct 2026 15:20:56 -0700 Subject: [PATCH 2/5] docs: document OpenRouter lanes and the any-model plan LANES.md gains an "OpenRouter: any model, one key" section, its env variables, keychain lines, price note, and a V6 route battery, and drops the "not shipped" note. README, USAGE, and LIVE-GATE name the new provider. UPSTREAM-FLEX lists the OpenRouter surfaces as fork-owned. docs/plans/2026-10-05-openrouter-gateway.md holds the plan with its runtime and setup flow diagrams. Refs #7 --- CHANGES.md | 7 + README.md | 13 +- UPSTREAM-FLEX.md | 1 + docs/LANES.md | 42 ++++- docs/LIVE-GATE.md | 2 +- docs/USAGE.md | 22 ++- docs/plans/2026-10-05-openrouter-gateway.md | 174 ++++++++++++++++++++ 7 files changed, 241 insertions(+), 20 deletions(-) create mode 100644 docs/plans/2026-10-05-openrouter-gateway.md diff --git a/CHANGES.md b/CHANGES.md index 9e791df2..e1fb45a1 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -1,5 +1,12 @@ # CHANGES — applied substitutions +## Unreleased: OpenRouter gateway, any model + +- New gateway provider `openrouter` ([#7](https://github.com/thisguymartin/pstack-flex/issues/7)). One `OPENROUTER_API_KEY` reaches any model in OpenRouter's catalog through the stock `claude` binary, the same path DeepSeek and MiniMax use. There is no allowlist: the flex matrix carries one open `openrouter` row, each distinct model ID is its own family, and setup's live probe on the named model is the gate. The OpenRouter probe reads its marker from a file, so it proves a tool call. +- The runner refuses an OpenRouter ID without a namespace and OpenRouter's own `openrouter/*` routers, which pick the model server-side. OpenRouter reports must match the requested ID exactly apart from case; the other gateways keep their prefix rule. Only OpenRouter lanes get the empty `ANTHROPIC_API_KEY` its guide requires, so DeepSeek and MiniMax environments are unchanged. +- Panel diversity counts the lab behind the model. An OpenRouter lane counts as its model ID's namespace, and `anthropic`, `openai`, `x-ai`, `deepseek`, and `minimax` match the direct providers. +- Design and flow diagrams: [docs/plans/2026-10-05-openrouter-gateway.md](docs/plans/2026-10-05-openrouter-gateway.md). + ## Unreleased: pstack-flex becomes its own distribution - The marketplace is now `pstack-flex` (was `open-pstack`) in both the Claude Code and Codex marketplace files. Install with `pstack@pstack-flex`. The plugin keeps the name `pstack`, so skill names such as `pstack:poteto-mode` are unchanged. Manifests, package names, and docs point at `thisguymartin/pstack-flex`; attribution to open-pstack, pstack-claude, and Cursor pstack stays in README and NOTICE. diff --git a/README.md b/README.md index 04e937ac..6d5c4a13 100644 --- a/README.md +++ b/README.md @@ -47,10 +47,11 @@ Every pstack role (who writes code, who explores, who sits on a review panel) ma | `deepseek-pro` | `deepseek:deepseek-v4-pro@high` | `DEEPSEEK_API_KEY` | none; selectable | | `minimax` | `minimax:MiniMax-M3@high` | `MINIMAX_API_KEY` | none; selectable | | `minimax-preview` | `minimax:MiniMax-M3.1-Flash-Preview@high` | `MINIMAX_API_KEY` (Token Plan) | none; selectable | +| `openrouter` | `openrouter:/@high`, any OpenRouter model | `OPENROUTER_API_KEY` | none; selectable | -The default review panel is `claude:fable@max, codex:gpt-6-astra@high, grok:grok-4.7@xhigh, claude:opus@max`: four lanes across three providers. Any family can take any role. Panels must span at least two providers, and two models from one provider count as one, because the adversarial signal comes from model diversity. +The default review panel is `claude:fable@max, codex:gpt-6-astra@high, grok:grok-4.7@xhigh, claude:opus@max`: four lanes across three providers. Any family can take any role. Panels must span at least two providers, and two models from one provider count as one, because the adversarial signal comes from model diversity. An OpenRouter lane counts as the lab that made its model. -The DeepSeek and MiniMax lanes run the stock `claude` binary against the lab's Anthropic-compatible endpoint with that lab's key, in an isolated config directory, with inherited Anthropic routing stripped. A lane refuses to start if it finds a claude.ai login in that directory, so a subscription credential can never reach a third-party endpoint. Their receipts keep real token usage but set `costUsd` to null (Claude Code prices at Anthropic rates); the price table is in [docs/LANES.md](docs/LANES.md). Anthropic does not support pointing Claude Code at non-Anthropic endpoints; use synthetic data for gateway testing and keep keys in your local environment. +The DeepSeek, MiniMax, and OpenRouter lanes run the stock `claude` binary against an Anthropic-compatible endpoint with that provider's key, in an isolated config directory, with inherited Anthropic routing stripped. A lane refuses to start if it finds a claude.ai login in that directory, so a subscription credential can never reach a third-party endpoint. Their receipts keep real token usage but set `costUsd` to null (Claude Code prices at Anthropic rates); the price table is in [docs/LANES.md](docs/LANES.md). Anthropic does not support pointing Claude Code at non-Anthropic endpoints; use synthetic data for gateway testing and keep keys in your local environment. Through OpenRouter, a role can use any model in its catalog, such as `openrouter:moonshotai/kimi-k3@high`; setup's live probe on that model is the only gate. ### How a role becomes a lane @@ -61,7 +62,7 @@ flowchart LR P -->|"any other provider"| R["pstack-runner
one process per lane"] R --> C1["codex CLI"] R --> C2["grok CLI"] - R --> C3["claude CLI + env
DeepSeek or MiniMax endpoint"] + R --> C3["claude CLI + env
DeepSeek, MiniMax, or OpenRouter endpoint"] N --> O["Output + receipt
model, effort, tokens, status"] C1 --> O C2 --> O @@ -72,7 +73,7 @@ The parent resolves every route once, before fan-out. Children never detect the ## Install -You need a current Claude Code or Codex installation and [Bun](https://bun.sh) for the lane runner. Sign in only to the CLIs whose plans you have (Claude Code, Codex, Grok), and export `DEEPSEEK_API_KEY` or `MINIMAX_API_KEY` in the shell that starts your session for the gateway lanes. Any subset works, down to a zero-subscription setup on two keys. +You need a current Claude Code or Codex installation and [Bun](https://bun.sh) for the lane runner. Sign in only to the CLIs whose plans you have (Claude Code, Codex, Grok), and export `DEEPSEEK_API_KEY`, `MINIMAX_API_KEY`, or `OPENROUTER_API_KEY` in the shell that starts your session for the gateway lanes. Any subset works, down to a zero-subscription setup on two keys. ### Claude Code @@ -179,10 +180,10 @@ Both apps read the same pstack skills. Only the way they start those skills and | Start poteto-mode | Run `/pstack:poteto-mode` or ask for pstack by name. A small startup instruction keeps Claude from starting pstack skills on its own. | Ask for `pstack:poteto-mode` by name. Codex runs the same startup instruction, so pstack skills also wait for a request there. | | Runs inside the app | Claude models stay inside Claude Code. | The Codex families stay inside Codex. | | Other models | The Codex families and Grok run through their signed-in command-line tools. | Claude and Grok run through their signed-in command-line tools. | -| Gateway models | DeepSeek and MiniMax always run through the external runner with an isolated config directory, never as a native agent. | Same. | +| Gateway models | DeepSeek, MiniMax, and OpenRouter always run through the external runner with an isolated config directory, never as a native agent. | Same. | | Skills and workflows | Shared with Codex. | Shared with Claude Code. | -Grok, DeepSeek, and MiniMax can take part in a multi-model review. You cannot use any of them as the main app running pstack. +Grok, DeepSeek, MiniMax, and OpenRouter models can take part in a multi-model review. You cannot use any of them as the main app running pstack. ## Upstream diff --git a/UPSTREAM-FLEX.md b/UPSTREAM-FLEX.md index b13e5a8c..8b6fa5f9 100644 --- a/UPSTREAM-FLEX.md +++ b/UPSTREAM-FLEX.md @@ -38,6 +38,7 @@ All flex changes are additive and live in port-owned files so upstream merges st - `docs/LANES.md`, this file, the README fork section, and the NOTICE/LICENSE/CHANGES additions - `plugins/pstack/hooks/session-start-context.md`, which the fork rewrote from open-pstack's auto-fire mandate into an opt-in gate, and the docs lines that describe it - The `intake` and `diff-behavior` skills: `plugins/pstack/skills/intake/` and `plugins/pstack/skills/diff-behavior/`. Upstream has no equivalent, so they never conflict. +- The OpenRouter gateway: its `GATEWAY_SPECS` row and `openRouterModelRefusal` in `runner/flex-providers.ts`, the router check in `run.ts` `validateOptions`, the exact-match branch in `parse-output.ts` `reportedModelMatches`, the open `openrouter` row and its paragraph in the flex section, the lab rule in the panel-diversity paragraph, and the OpenRouter lines in `skills/setup-pstack/SKILL.md`. - The lane journal: `runner/flex-journal.ts` and its test (new), its call sites in `runner/{types,run,cli}.ts` (an optional stdout callback on the model run, the journal opened after output reservation and finished on both return paths, and the `--label` flag), the journal tests in `run.test.ts`, and the pstack-flex paragraph after the invocation block in `references/provider-dispatch.md`. On a sync, keep these call sites; they change no receipt or exit status. Every upstream skill body is byte-unchanged except for the default-descriptor mentions listed above. Since [#17](https://github.com/thisguymartin/pstack-flex/issues/17), the stock matrix carries three fork-owned GPT-6 rows and the first-run sheet uses them, so those two surfaces conflict on every upstream sync and are resolved by hand: keep the fork's rows and defaults, take upstream's wording for everything else. diff --git a/docs/LANES.md b/docs/LANES.md index 36156629..b8947e2e 100644 --- a/docs/LANES.md +++ b/docs/LANES.md @@ -9,9 +9,9 @@ Prices and endpoints below were verified 2026-09-25 and drift. Re-verify against | Kind | Lanes | Auth | Billing | Route | | --- | --- | --- | --- | --- | | Subscription | `claude:fable`, `claude:opus`, `codex:gpt-6-astra`, `codex:gpt-6-sol`, `codex:gpt-6-luna`, `codex:gpt-5.6-sol`, `grok:grok-4.7` | each CLI's own login | that CLI's plan | native or external per the route table | -| Gateway (flex) | DeepSeek Flash / V4 Pro; MiniMax M3 / M3.1 Flash Preview | API key in the environment | provider billing; preview requires Token Plan | always the external runner | +| Gateway (flex) | DeepSeek Flash / V4 Pro; MiniMax M3 / M3.1 Flash Preview; any OpenRouter model | API key in the environment | provider billing; preview requires Token Plan | always the external runner | -A gateway lane is the stock `claude` binary env-pointed at the lab's Anthropic-compatible endpoint. There is no custom agent loop and no separate harness: the same runner that spawns Codex and Grok lanes spawns gateway lanes with injected environment. Both labs document this Claude Code setup themselves (DeepSeek: `deepseek-ai/awesome-deepseek-agent`, `docs/claude_code.md`; MiniMax: platform.minimax.io, Claude Code guide). +A gateway lane is the stock `claude` binary env-pointed at the lab's Anthropic-compatible endpoint. There is no custom agent loop and no separate harness: the same runner that spawns Codex and Grok lanes spawns gateway lanes with injected environment. Each provider documents this Claude Code setup itself (DeepSeek: `deepseek-ai/awesome-deepseek-agent`, `docs/claude_code.md`; MiniMax: platform.minimax.io, Claude Code guide; OpenRouter: [Claude Code integration](https://openrouter.ai/docs/guides/guides/claude-code-integration)). ## GPT-6 Codex families @@ -44,18 +44,39 @@ As of 2026-09-27, [MiniMax's model guide](https://platform.minimax.io/docs/guide Before recommending a fastest or strongest default, compare the same synthetic coding tasks for correctness, completion time, tool-call reliability, token usage, and actual provider billing. Preview pricing and plan limits must be checked against the active plan rather than inferred from M3 rates. +## OpenRouter: any model, one key + +One `OPENROUTER_API_KEY` reaches every model in [OpenRouter's catalog](https://openrouter.ai/models). There is no allowlist. Name any model ID with its namespace, and setup's live probe on that exact model is the gate: + +```text +openrouter:moonshotai/kimi-k3@high +openrouter:z-ai/glm-5.3@xhigh +openrouter:google/gemini-3.8-flash@low +``` + +`curl -s https://openrouter.ai/api/v1/models` lists the IDs without a key. Each distinct model ID is its own family with its own effort and probe. + +- **One refusal.** The runner refuses OpenRouter's own routers (`openrouter/auto`, `openrouter/free`, and the rest of the `openrouter/` namespace). They choose the model server-side, which would hide which model ran. Every model they could choose is reachable by its own ID. +- **Tools are required.** A lane is a Claude Code agent, so the model must support tool calls. Setup's OpenRouter probe reads its marker from a file, so a model without tool support fails there, not in a real lane. +- **What OpenRouter guarantees.** OpenRouter guarantees Claude Code only with Anthropic's first-party models. Other models work as far as their probe shows; [gateway-model-probes.md](gateway-model-probes.md) records the route evidence across labs. +- **Effort.** OpenRouter maps the requested effort onto each model's reasoning controls. A receipt proves the request, not the applied depth. +- **Model proof.** The reported model must match the requested ID exactly, apart from case. A sibling such as `z-ai/glm-5.3-air` fails a `z-ai/glm-5.3` lane. OpenRouter may fail over between hosts serving the same model; it does not swap the model unless you ask it to through a router or fallback list, which pstack never sends. +- **Context.** Claude Code does not know a third-party model's context window. When a model's window is small, a long lane can fail as it fills; `OPENROUTER_MAX_CONTEXT_TOKENS` sets the cap for every OpenRouter lane. +- **Panel diversity counts labs.** An OpenRouter lane counts as its model ID's namespace. `anthropic`, `openai`, `x-ai`, `deepseek`, and `minimax` match the `claude`, `codex`, `grok`, `deepseek`, and `minimax` providers. `openrouter:deepseek/deepseek-v4-pro` plus `deepseek:deepseek-flash` is one provider. +- **Privacy and spend live on OpenRouter's dashboard.** Turn off data collection or require zero-data-retention hosts, and set a credit limit on the key. OpenRouter forwards each prompt to whichever host serves the model. + ## Gateway environment reference Set by you: | Variable | Required | Meaning | | --- | --- | --- | -| `DEEPSEEK_API_KEY` / `MINIMAX_API_KEY` | yes, per lane | the lab's API key; the lane refuses to start without it | -| `DEEPSEEK_BASE_URL` / `MINIMAX_BASE_URL` | no | endpoint override; defaults are in the flex model matrix | -| `PSTACK_FLEX_DEEPSEEK_CONFIG_DIR` / `PSTACK_FLEX_MINIMAX_CONFIG_DIR` | no | config-dir override; default `~/.pstack-flex/` | -| `DEEPSEEK_MAX_CONTEXT_TOKENS` / `MINIMAX_MAX_CONTEXT_TOKENS` | no | context-cap override for the claude CLI | +| `DEEPSEEK_API_KEY` / `MINIMAX_API_KEY` / `OPENROUTER_API_KEY` | yes, per lane | the lab's API key; the lane refuses to start without it | +| `DEEPSEEK_BASE_URL` / `MINIMAX_BASE_URL` / `OPENROUTER_BASE_URL` | no | endpoint override; defaults are in the flex model matrix. OpenRouter's must end in `/api`, not the `/api/v1` other tools use | +| `PSTACK_FLEX_DEEPSEEK_CONFIG_DIR` / `PSTACK_FLEX_MINIMAX_CONFIG_DIR` / `PSTACK_FLEX_OPENROUTER_CONFIG_DIR` | no | config-dir override; default `~/.pstack-flex/` | +| `DEEPSEEK_MAX_CONTEXT_TOKENS` / `MINIMAX_MAX_CONTEXT_TOKENS` / `OPENROUTER_MAX_CONTEXT_TOKENS` | no | context-cap override for the claude CLI | -Injected by the runner at spawn time (never written to disk, never in receipts): `ANTHROPIC_BASE_URL`, `ANTHROPIC_AUTH_TOKEN`, the model pins (`ANTHROPIC_MODEL`, the opus/sonnet/haiku alias defaults, `CLAUDE_CODE_SUBAGENT_MODEL`), `CLAUDE_CODE_ATTRIBUTION_HEADER=0`, `CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1`, and `CLAUDE_CONFIG_DIR`. The runner first removes inherited `ANTHROPIC_*` values and Claude Code cloud-provider flags from the parent session. +Injected by the runner at spawn time (never written to disk, never in receipts): `ANTHROPIC_BASE_URL`, `ANTHROPIC_AUTH_TOKEN`, the model pins (`ANTHROPIC_MODEL`, the opus/sonnet/haiku alias defaults, `CLAUDE_CODE_SUBAGENT_MODEL`), `CLAUDE_CODE_ATTRIBUTION_HEADER=0`, `CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1`, `CLAUDE_CONFIG_DIR`, and for OpenRouter an empty `ANTHROPIC_API_KEY`, as its guide requires. The runner first removes inherited `ANTHROPIC_*` values and Claude Code cloud-provider flags from the parent session. ## Storing keys @@ -69,11 +90,13 @@ Recommended: your OS keychain, loaded on demand. # once per key — prompts for the value, nothing lands in shell history security add-generic-password -a "$USER" -s pstack-deepseek -w security add-generic-password -a "$USER" -s pstack-minimax -w + security add-generic-password -a "$USER" -s pstack-openrouter -w # in .zshrc: a function, not an export — keys enter env only when called pstack-keys() { export DEEPSEEK_API_KEY=$(security find-generic-password -a "$USER" -s pstack-deepseek -w) export MINIMAX_API_KEY=$(security find-generic-password -a "$USER" -s pstack-minimax -w) + export OPENROUTER_API_KEY=$(security find-generic-password -a "$USER" -s pstack-openrouter -w) } ``` @@ -81,7 +104,7 @@ Recommended: your OS keychain, loaded on demand. - **1Password CLI**: `op run --env-file=.env.tpl -- claude` injects the keys at process start with biometric unlock and exports nothing into the shell permanently. - **direnv**: fine for per-project scoping (gateway lanes are per-project opt-in anyway), but a raw `.envrc` is plaintext — have it call the keychain instead of holding the key. -Honest threat model: encryption at rest protects against dotfile repos, backups, and file theft. Once a key is in process env, any process running as your user can read it — the same exposure your CLI OAuth credential files already have. Keychain storage plus two ops controls is the right amount: **set spend caps on the DeepSeek and MiniMax dashboards** (the real blast-radius limiter) and rotate keys if a machine is ever compromised. +Honest threat model: encryption at rest protects against dotfile repos, backups, and file theft. Once a key is in process env, any process running as your user can read it — the same exposure your CLI OAuth credential files already have. Keychain storage plus two ops controls is the right amount: **set spend caps on the DeepSeek, MiniMax, and OpenRouter dashboards** (on OpenRouter, a credit limit on the key) (the real blast-radius limiter) and rotate keys if a machine is ever compromised. ## Prices (verified 2026-09-25 — re-check before budgeting) @@ -90,6 +113,7 @@ Honest threat model: encryption at rest protects against dotfile repos, backups, | DeepSeek V4.1-Flash (`deepseek-flash`) | $0.30 in / $1.20 out peak; $0.15 / $0.60 off-peak; cache hits near-free | Off-peak windows: 01:00-04:00 and 06:00-10:00 UTC on weekdays. The discount is automatic on DeepSeek's side; pstack-flex surfaces the window but never delays your work to hit it. MIT open weights. | | DeepSeek V4-Pro | $1.32 / $3.96 peak; half off-peak | Stronger model for hard lanes; assign it per role if wanted. | | MiniMax M3 (`MiniMax-M3`) | $0.30 / $1.20 at up to 512K input; higher above | 1M context. Custom community model license (irrelevant for API use). | +| OpenRouter (any model) | the serving provider's price, passed through with no markup | OpenRouter charges 5.5% when you buy credits by card ([FAQ](https://openrouter.ai/docs/faq), checked 2026-10-05). Each model's page lists its price. | | Claude / Codex / Grok subscription lanes | plan-dependent | Billed by each provider's plan, not per token here. | Gateway receipts always report `costUsd: null`: the claude CLI computes `total_cost_usd` at Anthropic list prices, which would be fiction for third-party traffic. Token usage in receipts is real — multiply it by the table above. @@ -126,7 +150,6 @@ Quality note: this trades peak capability for cost control. The hardest-task rol ## Optional lanes -- **OpenRouter (not shipped).** OpenRouter documents a direct Claude Code connection (`ANTHROPIC_BASE_URL=https://openrouter.ai/api`, `ANTHROPIC_AUTH_TOKEN` from `OPENROUTER_API_KEY`, and an explicitly empty `ANTHROPIC_API_KEY`), so no local translator is needed. It guarantees that route only for Anthropic models, so DeepSeek, MiniMax, and other catalog models through OpenRouter must pass the live probes in [issue #7](https://github.com/thisguymartin/pstack-flex/issues/7) (tool call, second turn, effort, model identity) before a lane ships. If it does, model it as another gateway provider in `flex-providers.ts`. Expect a credit fee on top of provider list prices; check OpenRouter's current pricing page. - **Local via Ollama (planned).** Ollama serves an Anthropic-compatible API since v0.14, so a `local` gateway provider pointed at it is the natural next lane: full compute control, zero per-token cost, your hardware. Not wired in yet. ## Adding a gateway provider @@ -147,3 +170,4 @@ Any lab that serves an Anthropic-compatible `/v1/messages` endpoint can become a - V3: run `claude auth status --json` inside a fresh flex config dir with `ANTHROPIC_AUTH_TOKEN` set and record the output here. On macOS, confirm whether `claude login` under an explicit `CLAUDE_CONFIG_DIR` writes `.credentials.json` or the Keychain. - V4: the zero-subscription walkthrough above, end to end, on a machine with no stored provider logins. - V5: OAuth guard live: `claude login` inside a scratch flex config dir, run a lane, confirm the refusal receipt, then delete that login. +- V6: OpenRouter route battery ([#7](https://github.com/thisguymartin/pstack-flex/issues/7)) on models from at least four labs, read-only, synthetic workspace. For each: the exact ID answers; a tool call reads a file marker that is not in the prompt; a second turn uses that result; reasoning tokens differ between `low` and `high`; the receipt's reported model matches OpenRouter's activity log. Capture OpenRouter's real errors for a wrong ID, a bad key, no credits, and a model without tools. Record everything in [gateway-model-probes.md](gateway-model-probes.md). diff --git a/docs/LIVE-GATE.md b/docs/LIVE-GATE.md index fc275573..3592c5a8 100644 --- a/docs/LIVE-GATE.md +++ b/docs/LIVE-GATE.md @@ -40,7 +40,7 @@ Run the rows that match the change. A change that touches the runner or the mode | A. Opt-in gate | In a fresh session, ask for a two-line fix without naming pstack. Then ask again with "Use pstack for this." | The first request runs no `pstack:` skill. The second enters `pstack:poteto-mode`. | | B. Setup | Run `/pstack:setup-pstack` (Claude Code) or `Use pstack:setup-pstack.` (Codex). Keep defaults or change one role. | Every assigned family probes `complete`, the sheet is written to `~/.claude/pstack-models.md` or `~/.codex/pstack-models.md`, and a failed probe writes nothing. | | C. Mixed panel | `Use pstack:interrogate on the last commit.` | Each configured reviewer returns, external lanes write receipts with `status: complete`, and any missing CLI shows as a named dropout, not a substitute. | -| D. Gateway lane | With `DEEPSEEK_API_KEY` or `MINIMAX_API_KEY` exported, assign one role to that family in setup and run it once. | The receipt shows `status: complete`, the requested model, and `costUsd: null`. | +| D. Gateway lane | With `DEEPSEEK_API_KEY`, `MINIMAX_API_KEY`, or `OPENROUTER_API_KEY` exported, assign one role to that family in setup (for OpenRouter, any model ID you name) and run it once. | The receipt shows `status: complete`, the requested model, and `costUsd: null`. | | E. Lane journal | `mkdir -p ~/.pstack-flex/lanes`, run one external lane (for example an interrogate with a Codex reviewer from Claude Code), then `rm -rf ~/.pstack-flex/lanes`. With [psf-monitor](https://github.com/thisguymartin/psf-monitor) installed, watch the lane on its page instead. | While it runs, the lane's directory holds `lane.json` and a growing `stream.jsonl`; after it ends, `receipt.json` matches the runner's receipt. With the directory removed, the next lane writes nothing there and its receipt is unchanged. | | F. intake | In a scratch repo with a real issue: `Use pstack:intake for #.` | A brief appears under `.pstack/intake/`, with one playbook, an observable exit condition, and open questions when the issue is vague. No product code changes. | | G. diff-behavior | In a scratch repo, make a branch that changes one scenario on purpose and one by accident. `Use pstack:diff-behavior on this branch.` | The report lists the accidental change as unintended and the deliberate one as intended, with evidence for both sides. | diff --git a/docs/USAGE.md b/docs/USAGE.md index 069b85b9..91209d3a 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -6,7 +6,7 @@ The walkthrough: what this plugin is, how work flows through it, how to set it u pstack is a plugin of engineering skills, playbooks, and small local tools for coding agents — not a model, not a service. You hand `poteto-mode` a task; it matches the task to a playbook, works the steps, and leaves evidence (diffs, runs, receipts) you can inspect instead of asking for trust. Its sharpest edge is multi-model adversarial review: several different model families challenge important work, because the adversarial signal comes from model diversity, not assigned personas. -pstack-flex adds one thing on top: **you choose the models and the compute**. Any subset of families works, and two open labs — DeepSeek and MiniMax — are first-class lanes on plain API keys, down to a zero-subscription setup. +pstack-flex adds one thing on top: **you choose the models and the compute**. Any subset of families works, and two open labs — DeepSeek and MiniMax — are first-class lanes on plain API keys, down to a zero-subscription setup. With one OpenRouter key, a role can use any model in OpenRouter's catalog. If you also use my [thisguyskills](https://github.com/thisguymartin/skills) collection: that repo decides **what** to build (shaping, spec, Linear, handoff) and its handoff ends with "Use `pstack:poteto-mode`" — which is exactly where this repo picks up. @@ -23,11 +23,13 @@ flowchart TD F --> N3["grok:grok-4.7
grok CLI (Grok sub)"] F --> G1["deepseek:deepseek-flash
runner + env -> DeepSeek API (key)"] F --> G2["minimax:MiniMax-M3
runner + env -> MiniMax API (key)"] + F --> G3["openrouter:any/model
runner + env -> OpenRouter API (key)"] N1 --> R[Outputs + receipts] N2 --> R N3 --> R G1 --> R G2 --> R + G3 --> R R --> V[Verification: run it, judge it,
cross-model consensus] V --> PR([Review-ready PR]) ``` @@ -57,21 +59,23 @@ Plus [Bun](https://bun.sh) for the lane runner, and `multi_agent = true` under ` ## Keys for the gateway lanes -DeepSeek and MiniMax have no login flow here; their lanes read an API key from your environment at spawn time. The runner never writes keys to disk or receipts, so the only question is how the env gets populated. Don't paste keys into `.zshrc` — store them encrypted and load on demand. macOS Keychain, built in and free: +DeepSeek, MiniMax, and OpenRouter have no login flow here; their lanes read an API key from your environment at spawn time. The runner never writes keys to disk or receipts, so the only question is how the env gets populated. Don't paste keys into `.zshrc` — store them encrypted and load on demand. macOS Keychain, built in and free: ```zsh # once: store each key (prompts for the value, nothing in shell history) security add-generic-password -a "$USER" -s pstack-deepseek -w security add-generic-password -a "$USER" -s pstack-minimax -w +security add-generic-password -a "$USER" -s pstack-openrouter -w # in .zshrc: a function, not an export — keys enter env only when you call it pstack-keys() { export DEEPSEEK_API_KEY=$(security find-generic-password -a "$USER" -s pstack-deepseek -w) export MINIMAX_API_KEY=$(security find-generic-password -a "$USER" -s pstack-minimax -w) + export OPENROUTER_API_KEY=$(security find-generic-password -a "$USER" -s pstack-openrouter -w) } ``` -Daily flow: `pstack-keys -> claude -> /pstack:poteto-mode`. Alternatives, the threat model, and the spend-cap advice are in [LANES.md](LANES.md#storing-keys). Set spend caps on both provider dashboards; that is the real blast-radius control. +Daily flow: `pstack-keys -> claude -> /pstack:poteto-mode`. Alternatives, the threat model, and the spend-cap advice are in [LANES.md](LANES.md#storing-keys). Set spend caps on each provider dashboard (on OpenRouter, a credit limit on the key); that is the real blast-radius control. ## First-time setup: /setup-pstack @@ -194,7 +198,7 @@ sequenceDiagram participant P as Parent session participant R as pstack-runner participant C as claude -p (subprocess) - participant D as DeepSeek / MiniMax API + participant D as DeepSeek / MiniMax / OpenRouter API P->>R: lane: deepseek:deepseek-flash@high R->>R: guard: DEEPSEEK_API_KEY set?
config dir free of OAuth creds? Note over R: refusal = unauthenticated receipt,
no subprocess ever spawned @@ -255,3 +259,13 @@ bug-fix: codex:gpt-5.6-sol@max ## Selecting the additional gateway models Run `/setup-pstack` and assign `deepseek-pro` (`deepseek:deepseek-v4-pro@high`) or `minimax-preview` (`minimax:MiniMax-M3.1-Flash-Preview@high`) to named roles. Existing `deepseek` and `minimax` choices remain available. Each model has its own effort selection and live probe. MiniMax preview requires an eligible Token Plan key in `MINIMAX_API_KEY`; see [model choices and thinking controls](LANES.md#multiple-models-per-provider). No existing assignment changes until setup succeeds and you confirm the rendered sheet. + +## Using any model through OpenRouter + +Export `OPENROUTER_API_KEY`, run `/setup-pstack`, and give a role any model ID from [OpenRouter's catalog](https://openrouter.ai/models), written with its namespace: + +```text +interrogate reviewers: claude:fable@max, openrouter:moonshotai/kimi-k3@high, openrouter:z-ai/glm-5.3@high +``` + +There is no list to pick from. Setup probes the exact model you name, with a marker the model has to read from a file, and writes nothing if the probe fails. The only refused IDs are OpenRouter's own routers (`openrouter/auto`, `openrouter/free`), because they choose the model for you. Panel diversity counts the lab behind the model, so the sheet above spans three providers. Turn off data collection on OpenRouter's privacy settings, or require zero-data-retention hosts, before sending real code. See [LANES.md](LANES.md#openrouter-any-model-one-key). diff --git a/docs/plans/2026-10-05-openrouter-gateway.md b/docs/plans/2026-10-05-openrouter-gateway.md new file mode 100644 index 00000000..c98b1a26 --- /dev/null +++ b/docs/plans/2026-10-05-openrouter-gateway.md @@ -0,0 +1,174 @@ +# Plan: OpenRouter gateway lanes, any model (issue #7) + +Status (2026-10-05): Steps 1, 3, and 4 are implemented on `feat/openrouter-gateway`. Step 2 (live route probes) and the live gate are pending. + +## Context + +Goal: assign any pstack role to **any model OpenRouter serves** through one OpenRouter key, and know that the model you picked is the one that ran. There is no allowlist. The setup probe on the model you pick is the only gate. If the model can't run, the lane fails loudly and nothing silently swaps to another model. + +**Which harness? None new.** An OpenRouter lane is the stock Claude Code CLI (`claude -p`), pointed at OpenRouter's Anthropic-compatible endpoint through environment variables. DeepSeek and MiniMax already run this way (the "gateway" path in `pstack-runner`). The parent harness stays Claude Code or Codex. No proxy, no OpenCode, no custom agent loop, no new runner. + +**One exclusion: OpenRouter's own router IDs** (`openrouter/auto`, `openrouter/free`, and the rest of the `openrouter/` namespace). They pick the model for you, which breaks the project rule against automatic model selection and fallback. Every model a router could pick is reachable directly by its own ID, so no model is lost. + +## Diagram 1: what runs where + +``` +Parent harness: Claude Code or Codex (unchanged) + model sheet → arena runners: claude:fable@max, openrouter:z-ai/glm-5.3@high + │ argv: --provider openrouter --model z-ai/glm-5.3 --effort high --mode read-only + ▼ +pstack-runner (existing gateway path) + 1 validate any namespaced OpenRouter ID; only openrouter/* routers refused + 2 guard OPENROUTER_API_KEY set? ~/.pstack-flex/openrouter free of claude.ai OAuth? + 3 env strip inherited ANTHROPIC_* → inject OpenRouter URL, token, model pins + 4 preflight claude --version + │ spawn + ▼ +Child harness: stock `claude -p` (Claude Code CLI, headless) + --model z-ai/glm-5.3 --effort high --permission-mode plan --output-format json + ANTHROPIC_BASE_URL=https://openrouter.ai/api ANTHROPIC_AUTH_TOKEN=$OPENROUTER_API_KEY + ANTHROPIC_API_KEY="" CLAUDE_CONFIG_DIR=~/.pstack-flex/openrouter + │ Anthropic Messages API + ▼ +OpenRouter /api/v1/messages (normalizes --effort into each model's reasoning setting) + account: no data collection (or ZDR-only hosts), credit limit on the key + may fail over between hosts of the SAME model; never swaps the model + ├──▶ Google ├──▶ Z.ai ├──▶ Moonshot ├──▶ Qwen ├──▶ … any of ~460 models + ▼ +pstack-runner + 5 parse claude JSON → text + token usage; costUsd = null + 6 prove reported model == requested (exact) → else the lane fails, no fallback + 7 receipt → parent drains it like any other lane +``` + +## Diagram 2: picking a model in /setup-pstack + +``` +operator: "use moonshotai/kimi-k3 for interrogate reviewers" + │ + ▼ +descriptor openrouter:moonshotai/kimi-k3@high + (each distinct OpenRouter model = its own family: own effort, own probe) + │ + ▼ +probe runner, read-only, one turn; the marker is in a FILE, not the prompt + → proves the model answers AND makes a tool call + │ + ├─ fail ─▶ named error, sheet unchanged + │ wrong ID / no endpoints → unavailable-model + │ bad key → unauthenticated + │ no credits / no tools → child-failed, with OpenRouter's message + ▼ pass +diversity lab = the ID's namespace (moonshotai); anthropic/openai/x-ai/deepseek/minimax + count as the same lab as claude/codex/grok/deepseek/minimax lanes + │ + ▼ +confirm → sheet written +``` + +## Operator flow (after it ships) + +1. Store the key in the keychain; `pstack-keys` exports `OPENROUTER_API_KEY` (same pattern as the DeepSeek/MiniMax keys in `docs/LANES.md`). +2. OpenRouter dashboard: turn off data collection (or require ZDR), set a credit limit on the key. +3. `/setup-pstack` → name any OpenRouter model ID for any role → choose an effort → the probe runs → confirm. + +## Implementation + +Branch `feat/openrouter-gateway` from `main`. All edits stay in port-only files. Upstream-derived skill bodies (arena, interrogate, poteto-mode `SKILL.md`) stay byte-unchanged, per `UPSTREAM-FLEX.md`. + +### Step 1: the provider (code + unit tests) + +All files are under `plugins/pstack/skills/poteto-mode/scripts/runner/`. + +- `types.ts`: add `"openrouter"` to `GATEWAY_PROVIDERS`. The guard, preflight, env, parser, and model proof all branch on `isGatewayProvider`, so no new `switch` case is needed. +- `flex-providers.ts`: add a `GATEWAY_SPECS` row: + - `OPENROUTER_API_KEY` + - `https://openrouter.ai/api` + - `OPENROUTER_BASE_URL` + - `PSTACK_FLEX_OPENROUTER_CONFIG_DIR` + - no context default + - `OPENROUTER_MAX_CONTEXT_TOKENS` + + Add one spec field so only OpenRouter gets `ANTHROPIC_API_KEY=""`. `run.ts` already strips the inherited key, so this changes "unset" to "empty" for OpenRouter alone, and the DeepSeek/MiniMax env stays byte-identical. +- `run.ts` `validateOptions`: for `openrouter`, require a namespaced ID (`/`) and refuse the `openrouter/` namespace. Nothing else is filtered; the probe decides. +- `parse-output.ts` `reportedModelMatches`: use exact, case-insensitive matching for `openrouter`. The current gateway rule also accepts `${wanted}-…`, so `z-ai/glm-5.3-air` would wrongly pass for `z-ai/glm-5.3`. +- Tests: + - `flex-providers.test.ts`: the spec, the env map, and the empty key only for OpenRouter. + - `run.test.ts` `gateway lanes`: missing key, OAuth refusal, a namespaced ID through argv and receipt, a near-miss reported model failing, and `openrouter/auto` refused. + - `parse-output.test.ts`: exact-match cases. + - `commands.test.ts` and `cli.test.ts`: provider lists. + +### Step 2: prove the route on a spread of models (needs your key; costs cents) + +This is evidence that the path works for non-Anthropic models in general. It is not a list. + +Run the full #7 battery through the Step 1 runner on 5 models from different labs: `anthropic/claude-sonnet-5.5` (the control OpenRouter guarantees), `google/gemini-3.8-flash`, `moonshotai/kimi-k3`, `z-ai/glm-5.3`, `qwen/qwen3.8-max-0902`. Each run is read-only, in a synthetic workspace. + +The battery: +- the exact ID answers; +- a tool call reads a file marker that is not in the prompt; +- a second turn uses that result (thinking blocks survive); +- reasoning tokens differ between low and high; +- the receipt's reported model matches what OpenRouter's activity log shows was served. + +Also capture: +- OpenRouter's real failure strings: a wrong ID, 402 insufficient credits, a bad key, and a model without tool support; +- one `~` rolling alias and one `:free` variant, to learn what the receipt reports for each; +- empty versus unset `ANTHROPIC_API_KEY`. + +Write the results to `docs/gateway-model-probes.md` and to #7. Then: +- teach `run.ts` `unavailableStatus` the captured strings (for example, "no endpoints found" → `unavailable-model`), with fixtures from the real output; +- if a `~` alias or `:free` variant reports a different string than requested, add one documented translation rule in `reportedModelMatches`, or refuse that form with a clear message if no exact rule is possible. + +### Step 3: the contracts + +- `references/provider-dispatch.md`: + - One **open** row in the flex matrix: provider `openrouter`, model ``, default `high`, selectable `low`…`max`, key `OPENROUTER_API_KEY`, URL `https://openrouter.ai/api`. + - A rule that each distinct OpenRouter model is its own family. + - The descriptor grammar: split at the first `:` and the last `@`. + - Panel diversity counts labs: for `openrouter` lanes, the lab is the ID's namespace, and `anthropic`, `openai`, `x-ai`, `deepseek`, `minimax` equal the `claude`, `codex`, `grok`, `deepseek`, `minimax` providers. + - Add `openrouter` to the `--provider` list, the gateway paragraph, the override variables, and the route table. +- `setup-pstack/SKILL.md`: + - accept any OpenRouter ID for any role, mapping it to the open row; + - an OpenRouter probe row that puts the marker in a file; + - the lab-based diversity wording; + - a privacy and credit-limit disclosure before the first OpenRouter probe; + - the description line. +- `references/codex-tools.md`: the diversity lines. +- `runner/model-matrix.test.ts`: + - the open row's placeholder allowed only for `openrouter`, plus gateway ordering; + - no `openrouter:` in the first-run sheet; + - both diversity-string assertions; + - "Different models sharing a provider count as one provider" stays true for direct providers. + +### Step 4: docs + +- `docs/LANES.md`: replace "OpenRouter (not shipped)". + - Add OpenRouter to the lane-kind table, the env reference (including the empty key), prices (no token markup, 5.5% fee on credit purchases), and privacy. + - Note that `OPENROUTER_BASE_URL` must end in `/api`, not `/api/v1`. + - Note that models with less than 200K context can fail on long lanes; `OPENROUTER_MAX_CONTEXT_TOKENS` caps it. +- `README.md`: the gateway table and the lane diagram. +- `docs/USAGE.md`: keys, an example sheet with OpenRouter models, and the gateway sequence diagram. +- `docs/LIVE-GATE.md`: row D gets `OPENROUTER_API_KEY`. +- `UPSTREAM-FLEX.md`: list the open row and the lab rule under what the fork owns. +- `CHANGES.md`: an Unreleased entry. + +## Verification + +1. `bash scripts/check.sh`: Bun tests, strict typecheck, manifests, static invariants, `claude plugin validate`. +2. Step 2 evidence, committed in `docs/gateway-model-probes.md`. +3. Live gate (`docs/LIVE-GATE.md`, rows A–C in both harnesses plus row D): install the exact candidate. + - From **Claude Code** and from **Codex**, run `/setup-pstack`, type in an OpenRouter model that is **not** among the Step 2 models, and watch the probe pass and the sheet get written. + - Run a read-only mixed panel (for example `claude:fable` + `openrouter:z-ai/glm-5.3`) and check the receipts: `complete`, `provider-report`, `costUsd: null`. + - Check that a wrong ID, `openrouter/auto`, and a missing key each fail without changing the sheet. + + Record one evidence block per harness in the PR. The PR stays a draft until this is done. + +## Follow-ups (separate issues, not this PR) + +- **psf-monitor:** the Setup page's lane regex (`src/setup.ts`) rejects `/`, so OpenRouter roles would disappear from that view. +- **Credential leak risk on all gateways:** `CLAUDE_CODE_OAUTH_TOKEN` is not stripped from gateway children (`flex-providers.ts` `GATEWAY_INHERITED_CONFLICTS`, `run.ts` `CLAUDE_IDENTITY`). A parent's claude.ai token could reach a third-party endpoint. It affects DeepSeek/MiniMax too, so it gets its own fix and live gate. + +## Out of scope + +OpenRouter presets, the auto router or model fallback lists, real cost from OpenRouter billing, per-model automatic context caps, OpenRouter as the parent session (works today via the zero-subscription env walkthrough, docs only), OpenCode (#68), the registry redesign (#103). From f6bad90f842b700d6e732eb9a6b1d2f752edc4b1 Mon Sep 17 00:00:00 2001 From: Martin Patino Date: Mon, 5 Oct 2026 15:22:23 -0700 Subject: [PATCH 3/5] docs: link the OpenCode and registry issues to open-pstack --- docs/plans/2026-10-05-openrouter-gateway.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/plans/2026-10-05-openrouter-gateway.md b/docs/plans/2026-10-05-openrouter-gateway.md index c98b1a26..90b58127 100644 --- a/docs/plans/2026-10-05-openrouter-gateway.md +++ b/docs/plans/2026-10-05-openrouter-gateway.md @@ -171,4 +171,4 @@ Write the results to `docs/gateway-model-probes.md` and to #7. Then: ## Out of scope -OpenRouter presets, the auto router or model fallback lists, real cost from OpenRouter billing, per-model automatic context caps, OpenRouter as the parent session (works today via the zero-subscription env walkthrough, docs only), OpenCode (#68), the registry redesign (#103). +OpenRouter presets, the auto router or model fallback lists, real cost from OpenRouter billing, per-model automatic context caps, OpenRouter as the parent session (works today via the zero-subscription env walkthrough, docs only), OpenCode ([open-pstack#68](https://github.com/ericlitman/open-pstack/issues/68)), the registry redesign ([open-pstack#103](https://github.com/ericlitman/open-pstack/issues/103)). From b1e56a18465504ed9efb73ee53736ebdd5a60742 Mon Sep 17 00:00:00 2001 From: Martin Patino Date: Mon, 5 Oct 2026 15:40:27 -0700 Subject: [PATCH 4/5] fix(runner): classify Claude Code's 401 result and read thinking tokens An OpenRouter dry run with an invalid key showed two gaps on Claude Code 2.1.289. A rejected key comes back as "Failed to authenticate. API Error: 401" with "api_error_status":401, which the unavailable-status pattern missed, so the lane was child-failed instead of unauthenticated. And usage.output_tokens_details.thinking_tokens was dropped; it is now recorded as reasoningTokens. Both apply to every claude-binary lane. Refs #7 --- .../scripts/runner/parse-output.test.ts | 14 ++++++++++++++ .../poteto-mode/scripts/runner/parse-output.ts | 4 +++- .../skills/poteto-mode/scripts/runner/run.test.ts | 13 +++++++++++++ .../skills/poteto-mode/scripts/runner/run.ts | 4 +++- 4 files changed, 33 insertions(+), 2 deletions(-) diff --git a/plugins/pstack/skills/poteto-mode/scripts/runner/parse-output.test.ts b/plugins/pstack/skills/poteto-mode/scripts/runner/parse-output.test.ts index bd4feabc..b8b2b53d 100644 --- a/plugins/pstack/skills/poteto-mode/scripts/runner/parse-output.test.ts +++ b/plugins/pstack/skills/poteto-mode/scripts/runner/parse-output.test.ts @@ -142,6 +142,20 @@ describe("parseProviderOutput", () => { expect(reportedModelMatches("claude", "MiniMax-M3", "minimax-m3")).toBe(false); }); + it("reads Claude Code's thinking tokens as reasoning tokens", () => { + const parsed = parseProviderOutput( + "openrouter", + JSON.stringify({ + result: "62", + usage: { input_tokens: 40, output_tokens: 900, output_tokens_details: { thinking_tokens: 870 } }, + modelUsage: { "z-ai/glm-5.3": {} }, + }), + "", + "z-ai/glm-5.3" + ); + expect(parsed.usage).toMatchObject({ outputTokens: 900, reasoningTokens: 870 }); + }); + it("matches OpenRouter models exactly, ignoring only case", () => { expect(reportedModelMatches("openrouter", "z-ai/glm-5.3", "z-ai/glm-5.3")).toBe(true); expect(reportedModelMatches("openrouter", "z-ai/glm-5.3", "Z-AI/GLM-5.3")).toBe(true); diff --git a/plugins/pstack/skills/poteto-mode/scripts/runner/parse-output.ts b/plugins/pstack/skills/poteto-mode/scripts/runner/parse-output.ts index dec15c17..6235baba 100644 --- a/plugins/pstack/skills/poteto-mode/scripts/runner/parse-output.ts +++ b/plugins/pstack/skills/poteto-mode/scripts/runner/parse-output.ts @@ -38,7 +38,9 @@ function normalizedUsage(value: unknown): NormalizedUsage | null { ), outputTokens: finiteNumber(usage.output_tokens), reasoningTokens: finiteNumber( - usage.reasoning_tokens ?? usage.reasoning_output_tokens + usage.reasoning_tokens ?? + usage.reasoning_output_tokens ?? + object(usage.output_tokens_details)?.thinking_tokens ), totalTokens: finiteNumber(usage.total_tokens), }; diff --git a/plugins/pstack/skills/poteto-mode/scripts/runner/run.test.ts b/plugins/pstack/skills/poteto-mode/scripts/runner/run.test.ts index 5b6d2cd7..36cdb33b 100644 --- a/plugins/pstack/skills/poteto-mode/scripts/runner/run.test.ts +++ b/plugins/pstack/skills/poteto-mode/scripts/runner/run.test.ts @@ -106,6 +106,11 @@ if (stage === "model" && process.env.FAKE_AUTH_ERROR === "1") { console.error("API error: authentication_error - invalid api key"); process.exit(1); } +if (stage === "model" && process.env.FAKE_AUTH_ERROR === "claude-401") { + // Claude Code 2.1.289's real result for a gateway key OpenRouter rejects. + console.log(JSON.stringify({type:"result",subtype:"success",is_error:true,api_error_status:401,result:"Failed to authenticate. API Error: 401 User not found.",modelUsage:{}})); + process.exit(1); +} if (process.env.FAKE_INVALID_MODEL === "1") { console.error("The requested model is not supported with this account."); process.exit(1); @@ -1270,6 +1275,14 @@ describe("gateway lanes", () => { expect(result.exitCode).toBe(77); expect(receipt(input.receiptPath).status).toBe("unauthenticated"); }); + + it("classifies Claude Code's own 401 result as unauthenticated", async () => { + process.env.FAKE_AUTH_ERROR = "claude-401"; + const input = gatewayOptions("openrouter", "claude-401"); + const result = await runLane(input); + expect(result.exitCode).toBe(77); + expect(receipt(input.receiptPath).status).toBe("unauthenticated"); + }); }); describe("childEnvironment", () => { diff --git a/plugins/pstack/skills/poteto-mode/scripts/runner/run.ts b/plugins/pstack/skills/poteto-mode/scripts/runner/run.ts index 8ec7aeb8..5561a38b 100644 --- a/plugins/pstack/skills/poteto-mode/scripts/runner/run.ts +++ b/plugins/pstack/skills/poteto-mode/scripts/runner/run.ts @@ -406,7 +406,9 @@ function successfulPreflightEvidence(provider: Provider, model: string): string } function unavailableStatus(value: string): ReceiptStatus { - if (/not logged in|unauthenticated|authentication|sign in|login required/i.test(value)) { + // Claude Code 2.1.289 reports a rejected gateway key as "Failed to + // authenticate. API Error: 401" with `"api_error_status":401` in its result. + if (/not logged in|unauthenticated|authenticat(e|ion)|sign in|login required|"api_error_status":\s*401\b/i.test(value)) { return "unauthenticated"; } if (/model.{0,40}(not found|unknown|unavailable|unsupported|not supported|invalid)|invalid.{0,20}model/i.test(value)) { From b2586228244b2b608675eb493da5a97001323bc3 Mon Sep 17 00:00:00 2001 From: Martin Patino Date: Mon, 5 Oct 2026 15:40:27 -0700 Subject: [PATCH 5/5] feat(scripts): add the OpenRouter route battery and dry-run evidence scripts/probe-openrouter.sh runs issue #7's checks through pstack-runner: a two-file tool chain whose value is not in the prompt, an effort comparison by reasoning tokens, OpenRouter's own view of the served model and host, and the failure strings for a wrong ID, a bad key, a router, a rolling alias, a :free variant, and a model without tools. It spends real credit, so it stays out of check.sh and CI. The key is read from the environment and never reaches an argument list. docs/gateway-model-probes.md records the invalid-key dry run: a 401 that Claude Code retries for about three minutes. LANES.md notes the slow failure and points V6 at the script. Refs #7 --- CHANGES.md | 2 + docs/LANES.md | 3 +- docs/gateway-model-probes.md | 27 +++++++ scripts/probe-openrouter.sh | 141 +++++++++++++++++++++++++++++++++++ 4 files changed, 172 insertions(+), 1 deletion(-) create mode 100644 scripts/probe-openrouter.sh diff --git a/CHANGES.md b/CHANGES.md index e1fb45a1..8381ac13 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -5,6 +5,8 @@ - New gateway provider `openrouter` ([#7](https://github.com/thisguymartin/pstack-flex/issues/7)). One `OPENROUTER_API_KEY` reaches any model in OpenRouter's catalog through the stock `claude` binary, the same path DeepSeek and MiniMax use. There is no allowlist: the flex matrix carries one open `openrouter` row, each distinct model ID is its own family, and setup's live probe on the named model is the gate. The OpenRouter probe reads its marker from a file, so it proves a tool call. - The runner refuses an OpenRouter ID without a namespace and OpenRouter's own `openrouter/*` routers, which pick the model server-side. OpenRouter reports must match the requested ID exactly apart from case; the other gateways keep their prefix rule. Only OpenRouter lanes get the empty `ANTHROPIC_API_KEY` its guide requires, so DeepSeek and MiniMax environments are unchanged. - Panel diversity counts the lab behind the model. An OpenRouter lane counts as its model ID's namespace, and `anthropic`, `openai`, `x-ai`, `deepseek`, and `minimax` match the direct providers. +- Runner fixes found by the OpenRouter dry run, which apply to every lane: Claude Code 2.1.289's 401 result ("Failed to authenticate", `"api_error_status":401`) now classifies as `unauthenticated` instead of `child-failed`, and `usage.output_tokens_details.thinking_tokens` is recorded as `reasoningTokens`. +- `scripts/probe-openrouter.sh` runs the #7 route battery (V6 in `docs/LANES.md`). It spends real credit, so it is not part of `check.sh` or CI. - Design and flow diagrams: [docs/plans/2026-10-05-openrouter-gateway.md](docs/plans/2026-10-05-openrouter-gateway.md). ## Unreleased: pstack-flex becomes its own distribution diff --git a/docs/LANES.md b/docs/LANES.md index b8947e2e..cd05b0e8 100644 --- a/docs/LANES.md +++ b/docs/LANES.md @@ -63,6 +63,7 @@ openrouter:google/gemini-3.8-flash@low - **Model proof.** The reported model must match the requested ID exactly, apart from case. A sibling such as `z-ai/glm-5.3-air` fails a `z-ai/glm-5.3` lane. OpenRouter may fail over between hosts serving the same model; it does not swap the model unless you ask it to through a router or fallback list, which pstack never sends. - **Context.** Claude Code does not know a third-party model's context window. When a model's window is small, a long lane can fail as it fills; `OPENROUTER_MAX_CONTEXT_TOKENS` sets the cap for every OpenRouter lane. - **Panel diversity counts labs.** An OpenRouter lane counts as its model ID's namespace. `anthropic`, `openai`, `x-ai`, `deepseek`, and `minimax` match the `claude`, `codex`, `grok`, `deepseek`, and `minimax` providers. `openrouter:deepseek/deepseek-v4-pro` plus `deepseek:deepseek-flash` is one provider. +- **A rejected key is slow to fail.** Claude Code 2.1.289 retries a 401 for about three minutes before it gives up. The receipt then says `unauthenticated`. - **Privacy and spend live on OpenRouter's dashboard.** Turn off data collection or require zero-data-retention hosts, and set a credit limit on the key. OpenRouter forwards each prompt to whichever host serves the model. ## Gateway environment reference @@ -170,4 +171,4 @@ Any lab that serves an Anthropic-compatible `/v1/messages` endpoint can become a - V3: run `claude auth status --json` inside a fresh flex config dir with `ANTHROPIC_AUTH_TOKEN` set and record the output here. On macOS, confirm whether `claude login` under an explicit `CLAUDE_CONFIG_DIR` writes `.credentials.json` or the Keychain. - V4: the zero-subscription walkthrough above, end to end, on a machine with no stored provider logins. - V5: OAuth guard live: `claude login` inside a scratch flex config dir, run a lane, confirm the refusal receipt, then delete that login. -- V6: OpenRouter route battery ([#7](https://github.com/thisguymartin/pstack-flex/issues/7)) on models from at least four labs, read-only, synthetic workspace. For each: the exact ID answers; a tool call reads a file marker that is not in the prompt; a second turn uses that result; reasoning tokens differ between `low` and `high`; the receipt's reported model matches OpenRouter's activity log. Capture OpenRouter's real errors for a wrong ID, a bad key, no credits, and a model without tools. Record everything in [gateway-model-probes.md](gateway-model-probes.md). +- V6: OpenRouter route battery ([#7](https://github.com/thisguymartin/pstack-flex/issues/7)), scripted as `OPENROUTER_API_KEY=... bash scripts/probe-openrouter.sh [model ...]`, on models from at least four labs, read-only, synthetic workspace. For each: the exact ID answers; a tool call reads a file marker that is not in the prompt; a second turn uses that result; reasoning tokens differ between `low` and `high`; the receipt's reported model matches OpenRouter's activity log. Capture OpenRouter's real errors for a wrong ID, a bad key, no credits, and a model without tools. Record everything in [gateway-model-probes.md](gateway-model-probes.md). diff --git a/docs/gateway-model-probes.md b/docs/gateway-model-probes.md index ab03514a..138ddfec 100644 --- a/docs/gateway-model-probes.md +++ b/docs/gateway-model-probes.md @@ -28,3 +28,30 @@ These single short probes establish authentication, model selection, and success ## Remaining release gate Install the exact candidate and run setup from both real Claude Code and Codex user surfaces. Verify independent model effort choices, per-model probes, mixed-provider panels, saved-sheet readback, and unchanged configuration on failed access. Record installed version, surface, action, and observed result before merge or rollout. Changing only the runner's `--parent` flag would not satisfy this gate. + +## OpenRouter + +Tracking: [issue #7](https://github.com/thisguymartin/pstack-flex/issues/7) and draft PR [#31](https://github.com/thisguymartin/pstack-flex/pull/31). The battery is `scripts/probe-openrouter.sh` (V6 in [LANES.md](LANES.md#live-validation-checklist-before-merge-or-rollout-real-keys-never-in-ci)). + +### Dry run with an invalid key (2026-10-05) + +- Candidate: branch `feat/openrouter-gateway` at `f6bad90`, run from source, not installed. Parent: a Claude Code session, runner invoked with `--parent claude`. CLI: Claude Code 2.1.289. +- Action: `OPENROUTER_API_KEY=sk-or-v1-invalid-dryrun bash scripts/probe-openrouter.sh z-ai/glm-5.3`. It was stopped after the three per-model lanes. + +| Case | Effort | Exit | Elapsed ms | Claude Code result | +| --- | --- | --- | --- | --- | +| chain | high | 1 | 189180 | `api_error_status: 401`, `"Failed to authenticate. API Error: 401 User not found."`, `duration_api_ms: 0` | +| low | low | 1 | 182891 | same | +| high | high | 1 | 178841 | same | + +Findings: + +- The wrong key reached OpenRouter as the bearer token and was refused with 401. No other credential was used. +- Claude Code retries the 401 for about three minutes before it exits. +- The runner labelled these lanes `child-failed`, because its pattern matched "authentication" but not "Failed to authenticate". Fixed on the branch: Claude Code's 401 result now classifies as `unauthenticated`. +- Claude Code logs `[claude-code:unrecognized_model]` for the OpenRouter ID and still sends the request. +- Claude Code reports `usage.output_tokens_details.thinking_tokens`. The runner now records it as `reasoningTokens`, which the battery's effort check reads. + +### Route battery with a real key + +Pending. diff --git a/scripts/probe-openrouter.sh b/scripts/probe-openrouter.sh new file mode 100644 index 00000000..8d70d2a2 --- /dev/null +++ b/scripts/probe-openrouter.sh @@ -0,0 +1,141 @@ +#!/usr/bin/env bash +# V6 in docs/LANES.md: the OpenRouter route battery from issue #7. +# It spends real OpenRouter credit (cents), so it never runs in CI or +# check.sh. The key comes from OPENROUTER_API_KEY and is never printed or +# put on a command line. +# +# OPENROUTER_API_KEY=... bash scripts/probe-openrouter.sh [model ...] +# +# Per model, through pstack-runner (read-only, synthetic workspace): +# chain read start.txt, follow it to a second file, return the value +# there. Neither the file name nor the value is in the prompt, so +# a pass proves tool calls across turns. +# effort the same no-tool puzzle at low and at high; compare the +# reasoning tokens Claude Code reports. +# identity one direct OpenRouter call: the model and host it served. +# Then the failure strings: a wrong ID, a bad key, a model without tools, a +# `~` rolling alias, and a `:free` variant. The bad-key lane takes about three +# minutes: Claude Code retries a 401 before it gives up. +set -uo pipefail + +repo="$(cd "$(dirname "$0")/.." && pwd)" +runner="$repo/plugins/pstack/skills/poteto-mode/scripts/runner/pstack-runner" +api="https://openrouter.ai/api/v1" + +if [ -z "${OPENROUTER_API_KEY:-}" ]; then + echo "OPENROUTER_API_KEY is not set" >&2 + exit 2 +fi +models=("$@") +if [ "${#models[@]}" -eq 0 ]; then + models=( + anthropic/claude-sonnet-5.5 + google/gemini-3.8-flash + moonshotai/kimi-k3 + z-ai/glm-5.3 + qwen/qwen3.8-max-0902 + ) +fi + +out="$(mktemp -d "${TMPDIR:-/tmp}/openrouter-probes.XXXXXX")" +rows="$out/rows.tsv" +: > "$rows" + +workspace() { + local ws="$out/ws-$1" clue value + clue="clue-$(openssl rand -hex 4).txt" + value="PSF-$(openssl rand -hex 6)" + mkdir -p "$ws" + printf 'The value is in the file %s in this directory.\n' "$clue" > "$ws/start.txt" + printf '%s\n' "$value" > "$ws/$clue" + printf '%s' "$value" > "$out/expected-$1" + printf '%s' "$ws" +} + +# lane +lane() { + local name="$1" model="$2" effort="$3" prompt="$4" cwd="$5" expected="$6" + local dir="$out/$name" + mkdir -p "$dir" + printf '%s\n' "$prompt" > "$dir/prompt.md" + "$runner" --parent claude --provider openrouter --model "$model" --effort "$effort" \ + --mode read-only --prompt "$dir/prompt.md" --cwd "$cwd" \ + --output "$dir/output.txt" --receipt "$dir/receipt.json" \ + --label "openrouter probe $name" >/dev/null 2>"$dir/stderr.txt" + local got="" check="-" + [ -f "$dir/output.txt" ] && got="$(tr -d '[:space:]' < "$dir/output.txt")" + if [ -n "$expected" ]; then + if [ "$got" = "$expected" ]; then check="match"; else check="mismatch"; fi + fi + if [ -f "$dir/receipt.json" ]; then + jq -r --arg case "$name" --arg check "$check" '[ + $case, .model, .effort, .status, (.reportedModel // "-"), + (.modelEvidence // "-"), $check, (.usage.outputTokens // "-"), + (.usage.reasoningTokens // "-"), + .elapsedMs, ((.error.message // "-") | gsub("[\n|]"; " ") | .[0:160]) + ] | @tsv' "$dir/receipt.json" >> "$rows" + else + # The runner refused before reserving a receipt (a usage error). + printf '%s\t%s\t%s\tusage-error\t-\t-\t-\t-\t-\t-\t%s\n' "$name" "$model" "$effort" \ + "$(tr '\n|' ' ' < "$dir/stderr.txt" | cut -c1-160)" >> "$rows" + fi +} + +chain_prompt="Read start.txt in the current directory and follow what it says. Reply with only the value you find, nothing else." +puzzle_prompt="Do not use any tools. How many prime numbers are there below 300? Reply with only the number." + +for model in "${models[@]}"; do + slug="${model//[^A-Za-z0-9]/-}" + ws="$(workspace "$slug")" + lane "chain-$slug" "$model" high "$chain_prompt" "$ws" "$(cat "$out/expected-$slug")" + lane "low-$slug" "$model" low "$puzzle_prompt" "$ws" "62" + lane "high-$slug" "$model" high "$puzzle_prompt" "$ws" "62" +done + +# Failure strings and edge forms. The catalog is public; no key needed. +catalog="$out/catalog.json" +curl -fsS "$api/models" -o "$catalog" +no_tools="$(jq -r '[.data[] | select((.supported_parameters // []) | index("tools") | not) + | select(.id | test(":") | not) | select((.pricing.prompt | tonumber? // 1) > 0) + | select((.pricing.prompt | tonumber? // 1) < 0.000001)][0].id // empty' "$catalog")" +free_tools="$(jq -r '[.data[] | select(.id | endswith(":free")) + | select((.supported_parameters // []) | index("tools"))][0].id // empty' "$catalog")" + +ws="$(workspace edge)" +expected="$(cat "$out/expected-edge")" +lane "wrong-id" "z-ai/glm-does-not-exist" high "$chain_prompt" "$ws" "$expected" +OPENROUTER_API_KEY="sk-or-v1-invalid-probe" lane "bad-key" "z-ai/glm-5.3" high "$chain_prompt" "$ws" "$expected" +lane "router" "openrouter/auto" high "$chain_prompt" "$ws" "$expected" +lane "alias" "~google/gemini-flash-latest" high "$chain_prompt" "$ws" "$expected" +[ -n "$no_tools" ] && lane "no-tools" "$no_tools" high "$chain_prompt" "$ws" "$expected" +[ -n "$free_tools" ] && lane "free" "$free_tools" high "$chain_prompt" "$ws" "$expected" + +# What OpenRouter itself says it served. The header goes through stdin so +# the key never reaches a process argument list. +served="$out/served.tsv" +: > "$served" +for model in "${models[@]}" "~google/gemini-flash-latest" ${free_tools:+"$free_tools"}; do + body="$(jq -nc --arg model "$model" \ + '{model: $model, max_tokens: 512, reasoning: {effort: "low"}, + messages: [{role: "user", content: "Reply with OK."}]}')" + printf 'header = "Authorization: Bearer %s"\n' "$OPENROUTER_API_KEY" | + curl -sS --config - -H "Content-Type: application/json" -d "$body" \ + "$api/chat/completions" > "$out/served-${model//[^A-Za-z0-9]/-}.json" + jq -r --arg requested "$model" '[$requested, (.model // "-"), (.provider // "-"), + ((.error.message // "-") | .[0:120])] | @tsv' \ + "$out/served-${model//[^A-Za-z0-9]/-}.json" >> "$served" +done + +{ + echo "| Case | Requested model | Effort | Status | Reported model | Evidence | Output check | Output tokens | Reasoning tokens | Elapsed ms | Error |" + echo "| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |" + sed 's/\t/ | /g; s/^/| /; s/$/ |/' "$rows" + echo + echo "| Requested model | OpenRouter served | Host | Error |" + echo "| --- | --- | --- | --- |" + sed 's/\t/ | /g; s/^/| /; s/$/ |/' "$served" +} > "$out/results.md" + +cat "$out/results.md" +echo +echo "receipts, outputs, and stderr: $out"