Skip to content

Add OpenRouter as a gateway for any model - #31

Merged
thisguymartin merged 6 commits into
mainfrom
feat/openrouter-gateway
Oct 6, 2026
Merged

thisguymartin merged 6 commits into
mainfrom
feat/openrouter-gateway

Conversation

@thisguymartin

@thisguymartin thisguymartin commented Oct 5, 2026 •

Copy link
Copy Markdown
Owner

Closes #7

What changed

OpenRouter becomes a third gateway provider, next to DeepSeek and MiniMax. No new harness: the lane is the stock claude -p binary pointed at https://openrouter.ai/api with OPENROUTER_API_KEY, an isolated config dir, and the empty ANTHROPIC_API_KEY that OpenRouter's Claude Code guide requires.

  • Any model, no allowlist. The flex matrix gets one open openrouter row. Any OpenRouter model ID works for any role, such as openrouter:moonshotai/kimi-k3@high. Each distinct ID is its own family with its own effort and probe, and setup's live probe on the named model is the gate. The OpenRouter probe reads its marker from a file, so a pass proves a tool call.
  • One refusal. The runner refuses an ID without a namespace, and OpenRouter's own openrouter/* routers (auto, free), which pick the model server-side. Every model a router could pick is reachable by its own ID.
  • Exact model proof. An OpenRouter report must match the requested ID apart from case. The other gateways keep their prefix rule, which would accept z-ai/glm-5.3-air for z-ai/glm-5.3.
  • Diversity counts labs. An OpenRouter lane counts as its ID's namespace; anthropic, openai, x-ai, deepseek, and minimax match the direct providers.
  • DeepSeek and MiniMax are unchanged. Only OpenRouter gets the empty API key; their env maps stay byte-identical (the exact-map tests still pass).

The plan, with the runtime and setup flow diagrams, is in docs/plans/2026-10-05-openrouter-gateway.md.

Two runner fixes came out of an invalid-key dry run, and they apply to every claude-binary lane. Claude Code 2.1.289's 401 result ("Failed to authenticate", "api_error_status":401) now classifies as unauthenticated instead of child-failed. usage.output_tokens_details.thinking_tokens is now recorded as reasoningTokens. Evidence is in docs/gateway-model-probes.md.

scripts/probe-openrouter.sh runs the #7 route battery. It spends real credit, so it stays out of check.sh and CI.

Review order:

  1. runner/flex-providers.ts
  2. run.ts: validateOptions and unavailableStatus
  3. parse-output.ts: reportedModelMatches and normalizedUsage
  4. references/provider-dispatch.md and setup-pstack/SKILL.md
  5. the script and the docs

Still to do before this leaves draft

  • Step 2 of the plan: bash scripts/probe-openrouter.sh with a real key, on five models from different labs (V6 in docs/LANES.md). This also captures OpenRouter's real error strings so unavailableStatus can classify "no endpoints found" as unavailable-model (today it would be child-failed).
  • The live gate: rows A–D from docs/LIVE-GATE.md in Claude Code and Codex, with an OpenRouter model typed into setup.

Verification

  • Bun tests, strict typecheck, static invariants, and plugin validation pass: bash scripts/check.sh reports all checks passed at b258622. One test is timing-sensitive under load: runLane > spends one explicit deadline across preflight and model execution failed 5 of 5 runs on unchanged main on this machine in an earlier run. It does not touch the gateway path.
  • The exact candidate is installed in every affected harness.
  • The changed behavior passes from each real user surface.
  • The installed version, action, and observed result appear below.

Live evidence:

Pending.

A pull request without live evidence remains a draft. Do not merge, tag, release, or roll it out.

OpenRouter joins DeepSeek and MiniMax as a gateway provider: the stock
claude binary runs against https://openrouter.ai/api with the operator's
OPENROUTER_API_KEY, an isolated config dir, and the empty
ANTHROPIC_API_KEY that OpenRouter's Claude Code guide requires. Only
OpenRouter gets the empty key, so DeepSeek and MiniMax env is unchanged.

There is no model allowlist. The flex matrix carries one open
openrouter row; each distinct OpenRouter model ID is its own family,
and setup's live probe on the named model is the gate. The OpenRouter
probe reads its marker from a file, so a pass proves a tool call.

The runner refuses an ID without a namespace and OpenRouter's own
openrouter/* routers, which pick the model server-side. OpenRouter
reports must match the requested ID exactly apart from case, since a
prefix rule would accept a sibling model. Panel diversity counts the
lab behind the model: an OpenRouter lane counts as its ID's namespace.

Refs #7
LANES.md gains an "OpenRouter: any model, one key" section, its env
variables, keychain lines, price note, and a V6 route battery, and drops
the "not shipped" note. README, USAGE, and LIVE-GATE name the new
provider. UPSTREAM-FLEX lists the OpenRouter surfaces as fork-owned.
docs/plans/2026-10-05-openrouter-gateway.md holds the plan with its
runtime and setup flow diagrams.

Refs #7
An OpenRouter dry run with an invalid key showed two gaps on Claude Code
2.1.289. A rejected key comes back as "Failed to authenticate. API
Error: 401" with "api_error_status":401, which the unavailable-status
pattern missed, so the lane was child-failed instead of unauthenticated.
And usage.output_tokens_details.thinking_tokens was dropped; it is now
recorded as reasoningTokens. Both apply to every claude-binary lane.

Refs #7
scripts/probe-openrouter.sh runs issue #7's checks through
pstack-runner: a two-file tool chain whose value is not in the prompt,
an effort comparison by reasoning tokens, OpenRouter's own view of the
served model and host, and the failure strings for a wrong ID, a bad
key, a router, a rolling alias, a :free variant, and a model without
tools. It spends real credit, so it stays out of check.sh and CI. The
key is read from the environment and never reaches an argument list.

docs/gateway-model-probes.md records the invalid-key dry run: a 401
that Claude Code retries for about three minutes. LANES.md notes the
slow failure and points V6 at the script.

Refs #7
@thisguymartin

Copy link
Copy Markdown
Owner Author

How to test

This PR stays a draft until both parts below pass. Record the results in the evidence block at the end.

1. Route probes (script, real key)

Store the key once, in your own terminal (it prompts for the value):

security add-generic-password -a "$USER" -s pstack-openrouter -w

On OpenRouter's site, add a few dollars of credit, set a credit limit on the key, and turn off data collection in the privacy settings.

Run the battery from the branch checkout. It takes about 10 minutes and costs a few cents:

OPENROUTER_API_KEY=$(security find-generic-password -a "$USER" -s pstack-openrouter -w) \
  bash scripts/probe-openrouter.sh

Commit the printed tables to docs/gateway-model-probes.md under "Route battery with a real key". The bad-key case takes about three minutes, because Claude Code retries a 401.

2. Live gate (both apps)

Install this branch in place of any older open-pstack install:

# Claude Code
claude plugin uninstall pstack@open-pstack
claude plugin marketplace remove open-pstack
git clone -b feat/openrouter-gateway https://github.com/thisguymartin/pstack-flex ~/src/pstack-flex-candidate
claude plugin marketplace add ~/src/pstack-flex-candidate
claude plugin install pstack@pstack-flex

# Codex
codex plugin remove pstack@open-pstack
codex plugin marketplace remove open-pstack
codex plugin marketplace add thisguymartin/pstack-flex --ref feat/openrouter-gateway
codex plugin add pstack@pstack-flex

Start each app with the key loaded, so OpenRouter lanes can see it:

export OPENROUTER_API_KEY=$(security find-generic-password -a "$USER" -s pstack-openrouter -w)
claude      # and `codex` in a second terminal, same export first

In a new session in each app:

# Do this It passes when
A Ask for a tiny fix without mentioning pstack. Then ask again, adding "Use pstack for this." The first does not start pstack; the second does.
B Run /pstack:setup-pstack (in Codex: Use pstack:setup-pstack.). Change interrogate reviewers to claude:fable@max, openrouter:mistralai/mistral-medium-3-5@high. Any model works; this one is outside the probe script's list. Setup warns about privacy and credit, the OpenRouter probe completes, and the sheet is saved.
C Use pstack:interrogate on the last commit. Both reviewers return. The OpenRouter receipt shows status: complete, modelEvidence: provider-report, costUsd: null.
D Back in setup, try openrouter:openrouter/auto@high, then openrouter:z-ai/not-a-model@high. Both are refused or fail with a clear message, and the saved sheet doesn't change.

Evidence

One block per app, in the PR description under "Live evidence" (format from docs/LIVE-GATE.md):

Harness: Claude Code <claude --version> | Codex <codex --version>
Installed: pstack <version> @ <commit>
Row <letter>: <action you took>
Observed: <what happened, with receipt or screenshot path>
Result: pass | fail

@thisguymartin
thisguymartin marked this pull request as ready for review October 6, 2026 00:53
@thisguymartin
thisguymartin merged commit b49b15d into main Oct 6, 2026
1 check passed
@thisguymartin
thisguymartin deleted the feat/openrouter-gateway branch October 6, 2026 05:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add OpenRouter as a gateway for pstack model roles

1 participant