diff --git a/CHANGELOG.md b/CHANGELOG.md index e078b7d..b651a39 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,15 @@ +## 0.8.0 + +### Added + +- A `commandcode` backend for the same Jev decisions protocol: `createTypeSafe({ backend: "commandcode" })` sends judgments to `api.commandcode.ai` under `/provider/v1/systemone` with the key from `COMMANDCODE_API_KEY` and the model `typesafe/jev`; its public model list does not verify a key. +- `backend` accepts a caller-supplied endpoint object wherever a backend name is accepted (`createTypeSafe`, `keySituation`, `resolveApiKey`, `authState`, `ensureApiKey`, `safeError`): an endpoint names its own `label`, `host`, `keyEnv`, and optionally `path`, `defaultModel`, and model-list fields, is validated on every call, never reads `TYPESAFE_API_KEY` or the login store, and is never added to the registry. An invalid backend makes `authState` and `keySituation` throw `configuration`; validate user input with `resolveBackend` first. +- `resolveBackend(nameOrEndpoint)` resolves either form to the validated backend the client uses, and `backendHost(nameOrEndpoint)` reports the destination host for consent text, alongside the `BackendEndpoint`, `BackendSpec`, and `ResolvedBackend` types. +- `TypeSafeBackend` now includes `"commandcode"`; a consumer with an exhaustive `switch` over it sees a new member. + ## 0.7.4 ### Fixed diff --git a/README.md b/README.md index 03b1375..baed559 100644 --- a/README.md +++ b/README.md @@ -140,7 +140,7 @@ if (!answer.ok) return { skipped: answer.errorCode === "budget" }; // never thr Your extension owns its own user consent and budget; `/typesafe enable` applies only to this package's tool. Check `authState()` rather than your own consent flag before you report that judgments are on. -Judgments can also go through OpenRouter: `createTypeSafe({ backend: "openrouter" })` sends them to `openrouter.ai` with the key from `OPENROUTER_API_KEY`. That backend has no login store, so `/typesafe login` does not apply to it. Pass the same `backend` to `authState`, `keySituation`, and `ensureApiKey`, or the status you report describes the TypeSafe key while the requests use another one. The `/typesafe` commands and the `typesafe_evaluate` tool always use the TypeSafe backend. Every export — the client, `ask`, batching, the usage ledger, auth state, and the `pi-typesafe/calibrate` and `pi-typesafe/ui` entry points — is in [docs/api.md](docs/api.md). +Judgments can also go through OpenRouter: `createTypeSafe({ backend: "openrouter" })` sends them to `openrouter.ai` with the key from `OPENROUTER_API_KEY`. Command Code serves the same protocol: `createTypeSafe({ backend: "commandcode" })` sends them to `api.commandcode.ai` with the key from `COMMANDCODE_API_KEY` and the model `typesafe/jev`. Or pass a `BackendEndpoint` object as `backend` to name any host that serves the protocol: it brings its own key variable, never reads `TYPESAFE_API_KEY`, and `backendHost()` reports its destination host for your consent text. These backends have no login store, so `/typesafe login` does not apply to them. Pass the same `backend` to `authState`, `keySituation`, and `ensureApiKey`, or the status you report describes the TypeSafe key while the requests use another one. The `/typesafe` commands and the `typesafe_evaluate` tool always use the TypeSafe backend. Every export — the client, `ask`, batching, the usage ledger, auth state, and the `pi-typesafe/calibrate` and `pi-typesafe/ui` entry points — is in [docs/api.md](docs/api.md). ## Development diff --git a/docs/api.md b/docs/api.md index 79444f1..de5b8cc 100644 --- a/docs/api.md +++ b/docs/api.md @@ -24,8 +24,8 @@ result.answers.severity.score; // 0..2, may be fractional | Option | Default | Meaning | | --- | --- | --- | | `apiKey` | the backend's key (below) | Never returned | -| `backend` | `typesafe` | `typesafe` or `openrouter`; picks the host, the request path, the default model, and the key | -| `model` | `jev-latest` (`typesafe/jev-1.13` on OpenRouter) | No model is inferred from submitted content | +| `backend` | `typesafe` | `typesafe`, `openrouter`, `commandcode`, or a caller-supplied endpoint object; picks the host, the request path, the default model, and the key | +| `model` | `jev-latest` (`typesafe/jev-1.13` on OpenRouter, `typesafe/jev` on Command Code) | No model is inferred from submitted content | | `timeoutMs` | `15000` | Per request; no automatic retries | | `maxInputBytes` | `65536` | UTF-8 JSON bytes, not tokens | | `maxRequests` | `20` | Attempts per client instance, failures included | @@ -34,11 +34,29 @@ result.answers.severity.score; // 0..2, may be fractional | `ledger` | the store next to the key | Inject a ledger in tests | | `fetch` | global fetch | Inject a transport for offline tests | -A `model` is mapped to the backend's own id form before it is sent: on OpenRouter a bare `jev-latest` goes as `~typesafe/jev-latest` and a bare `jev-1.13` (or `jev-1.13.0`) as `typesafe/jev-1.13`, while an id that already carries an author, such as `vendor/other`, passes unchanged, and TypeSafe sends ids as written. The same mapping applies to a per-request `model` inside `evaluate()`; the limits of 1–100 characters apply to your own id, before mapping. +A `model` is mapped to the backend's own id form before it is sent: on OpenRouter a bare `jev-latest` goes as `~typesafe/jev-latest` and a bare `jev-1.13` (or `jev-1.13.0`) as `typesafe/jev-1.13`, while an id that already carries an author, such as `vendor/other`, passes unchanged, and TypeSafe sends ids as written. The same mapping applies to a per-request `model` inside `evaluate()`; the limits of 1–100 characters apply to your own id, before mapping. A caller-supplied endpoint's model is never mapped: `defaultModel` and a per-request `model` go on the wire as the caller wrote them. -`DECISIONS_BACKENDS` is the registry behind `backend`: each entry carries `label`, `host`, `keyEnv`, and, when the service does not serve the SDK's own paths, `path` for the judgment request plus `modelsPath`, `modelsField`, and `modelsIdField` for the model list — OpenRouter's list arrives under `data` and is renamed to the `models` the SDK reads, with each entry's `id` promoted to the `name` that `listModels()` returns. `modelsVerifyKey: false` marks a backend whose model list is public, and therefore proves nothing about the key. `DEFAULT_BACKEND` is `"typesafe"`. The TypeSafe backend takes its key from `TYPESAFE_API_KEY`, then the `/typesafe login` store. Every other backend reads only its own environment variable (`OPENROUTER_API_KEY` for OpenRouter): the store holds a TypeSafe key, and a login verifies against api.typesafe.ai, so neither applies elsewhere. Pass the same `backend` to `authState`, `keySituation`, and `ensureApiKey` so what you report matches what you send. +`DECISIONS_BACKENDS` is the registry behind `backend`: `typesafe`, `openrouter`, and `commandcode`. Command Code serves the same Jev decisions protocol at `api.commandcode.ai` under `/provider/v1/systemone`, with the model id `typesafe/jev`; its model list is public, under `/provider/v1/models`, and arrives in `data` with ids in `id`. Each entry carries `label`, `host`, `keyEnv`, and, when the service does not serve the SDK's own paths, `path` for the judgment request plus `modelsPath`, `modelsField`, and `modelsIdField` for the model list — OpenRouter's and Command Code's lists are renamed to the `models` the SDK reads, with each entry's `id` promoted to the `name` that `listModels()` returns. `modelsVerifyKey: false` marks a backend whose model list is public, and therefore proves nothing about the key: both OpenRouter and Command Code serve theirs without checking one, so `listModels()` there leaves the auth state unverified while the answer check is unchanged and a malformed reply stays a `response` error. `DEFAULT_BACKEND` is `"typesafe"`. -`evaluate(request, { signal })` validates before sending and rejects with `TypeSafeIntegrationError`. `code` is one of `configuration`, `validation`, `budget`, `aborted`, `timeout`, `http`, `connection`, `response`; messages never contain upstream bodies, keys, or your submitted state, and no header value except a numeric `Retry-After` count in seconds (quoted by the 429 advice as `Retry after seconds.`). The advice is backend-aware: a 401 says `Check TYPESAFE_API_KEY.` or `Check OPENROUTER_API_KEY.`, and a 402 says `Check your account balance.` except on OpenRouter, which says `Insufficient credits. Add credits at https://openrouter.ai/credits.` `listModels()` verifies the key without counting toward `maxRequests`, except on a backend whose model list is public (`modelsVerifyKey: false`), which accepts any key and leaves the auth state unverified. +`backend` also accepts a caller-supplied endpoint object (`BackendEndpoint`) wherever a backend name is accepted — `createTypeSafe`, `keySituation`, `resolveApiKey`, `authState`, `ensureApiKey`, and `safeError`. An endpoint names `label`, `host`, and `keyEnv`, and optionally `path`, `modelsPath`, `modelsField`, `modelsIdField`, `modelsVerifyKey`, and `defaultModel`; it is validated on every call, never added to the registry, and without `defaultModel` requires `model` on `createTypeSafe`. `resolveBackend(nameOrEndpoint)` resolves either form to the validated `ResolvedBackend` the client uses (registry `name` when there is one, `host` as origin only, explicit `keyEnv` and `modelsVerifyKey`), and `backendHost(nameOrEndpoint)` returns just the destination host — `api.commandcode.ai`, `gw.example.com:8443` — for consent text. Validation is in this order, and every failure is a `configuration` error whose message never quotes a caller value (a host can carry credentials in its user info), except the label after it is validated: + +| Rule | Refusal | +| --- | --- | +| `label` is a string, trimmed nonempty, at most 60 characters | `Backend label must be a nonempty string of at most 60 characters.` | +| `host` is an absolute `https:` URL with no user info, path, query, or fragment (`http:` only for a loopback host) | `Backend host must be an absolute https: URL with no user info, path, query, or fragment …` | +| `path`, when present, starts with `"/"` and carries no `?` or `#` | `Backend path must be a string that starts with "/".` | +| `modelsPath`, same rule | `Backend modelsPath must be a string that starts with "/".` | +| `modelsField` / `modelsIdField`, when present, nonempty strings | `Backend modelsField must be a nonempty string.` / `Backend modelsIdField must be a nonempty string.` | +| `modelsVerifyKey`, when present, a boolean | `Backend modelsVerifyKey must be a boolean.` | +| `keyEnv` is a name of letters, digits, and underscores, not starting with a digit | `Backend keyEnv must name an environment variable: letters, digits, and underscores, not starting with a digit.` | +| `keyEnv` is not `TYPESAFE_API_KEY` in any case | `Backend keyEnv must not be TYPESAFE_API_KEY: the TypeSafe key is only sent to the typesafe backend. Give this endpoint its own variable.` | +| `defaultModel`, when present, trimmed nonempty, at most 100 characters | `Backend defaultModel must be a nonempty string of at most 100 characters.` | + +Anything that is neither a registry name nor such an object is refused with `backend must be a registry name or a backend object.` + +The TypeSafe backend takes its key from `TYPESAFE_API_KEY`, then the `/typesafe login` store. Every other backend — registry or endpoint — reads only its own `keyEnv` variable: the store holds a TypeSafe key, and a login verifies against api.typesafe.ai, so neither applies elsewhere, and the TypeSafe key is only ever sent to the typesafe backend. `createTypeSafe` with no `apiKey` and no key in the backend's variable fails with `No API key. Set in the environment.` before any request is built. Pass the same `backend` to `authState`, `keySituation`, and `ensureApiKey` so what you report matches what you send. + +`evaluate(request, { signal })` validates before sending and rejects with `TypeSafeIntegrationError`. `code` is one of `configuration`, `validation`, `budget`, `aborted`, `timeout`, `http`, `connection`, `response`; messages never contain upstream bodies, keys, or your submitted state, and no header value except a numeric `Retry-After` count in seconds (quoted by the 429 advice as `Retry after seconds.`). The advice is backend-aware: a 401 names the backend's own key variable (`Check TYPESAFE_API_KEY.`, `Check OPENROUTER_API_KEY.`, `Check COMMANDCODE_API_KEY.`, or `Check .` for an endpoint), and a 402 says `Check your account balance.` except on OpenRouter, which says `Insufficient credits. Add credits at https://openrouter.ai/credits.` `listModels()` verifies the key without counting toward `maxRequests`, except on a backend whose model list is public (`modelsVerifyKey: false`, or an endpoint that does not set `modelsVerifyKey: true`), which accepts any key and leaves the auth state unverified. ## Admission @@ -75,7 +93,7 @@ The environment may lower an explicit cap, never raise it. A reached cap raises ## Auth state -`authState({ backend })` never throws. It reports `backend`, `kind` (`environment`, `stored`, `missing`, `unusable`), `keyName`, `path`, `reason`, `verified`, `verifiedAt`, `lastFailure`, and `usable` — `usable` is false when no key is present or the last authentication outcome was an HTTP 401/403 rejection. `backend` defaults to `typesafe`; name the backend you pass to `createTypeSafe`, or the report describes a key you do not send. The verification and failure record is one file shared by every backend, so after switching backends the last outcome stands until the next request. +`authState({ backend })` never throws for a valid backend. An invalid backend — an unknown name or an endpoint object that fails validation — throws the same `configuration` error as `resolveBackend`, so validate a user-supplied endpoint with `resolveBackend` first. It reports `backend` (the value you passed, `typesafe` by default — a name or your endpoint object), `kind` (`environment`, `stored`, `missing`, `unusable`), `keyName`, `path`, `reason`, `verified`, `verifiedAt`, `lastFailure`, and `usable` — `usable` is false when no key is present or the last authentication outcome was an HTTP 401/403 rejection. Name the backend you pass to `createTypeSafe`, or the report describes a key you do not send. The verification and failure record is one file shared by every backend, so after switching backends the last outcome stands until the next request: a 401 recorded while one backend is in use makes every backend's `authState` report `usable: false`. `describeAuth(state)` turns that into `{ level: "ok" | "warning" | "error", text }` for a status line or a log. The extension calls both at session start and after a rejection, so an enabled-but-unusable setup is never reported as working. diff --git a/package-lock.json b/package-lock.json index 07933aa..ff92ae6 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "pi-typesafe", - "version": "0.7.4", + "version": "0.8.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "pi-typesafe", - "version": "0.7.4", + "version": "0.8.0", "license": "MIT", "dependencies": { "@typesafe-ai/sdk": "^0.6.0", diff --git a/package.json b/package.json index 00d0377..0fe61af 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "pi-typesafe", - "version": "0.7.4", + "version": "0.8.0", "description": "TypeSafe AI (Jev) decisions for Pi: batched Choice/Score/Noul evaluation tool, terminal playground, and a typed API other extensions build on.", "type": "module", "license": "MIT", diff --git a/src/auth.ts b/src/auth.ts index 8a11de2..4f06377 100644 --- a/src/auth.ts +++ b/src/auth.ts @@ -1,7 +1,7 @@ import { chmodSync, mkdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs"; import { join } from "node:path"; -import { DEFAULT_BACKEND, TYPESAFE_KEY_ENV, backendConfig, usesTypesafeKey } from "./backends.js"; -import type { TypeSafeBackend } from "./backends.js"; +import { DEFAULT_BACKEND, TYPESAFE_KEY_ENV, resolveBackend, usesTypesafeKey } from "./backends.js"; +import type { BackendSpec } from "./backends.js"; import { credentialsPath, keySituation, keySourceLabel, piTypesafeDir } from "./credentials.js"; import type { KeySource } from "./credentials.js"; import { TypeSafeIntegrationError } from "./errors.js"; @@ -28,8 +28,8 @@ export interface AuthFailure { * that judgments will happen — an enabled extension with no key used to look identical to a working one. */ export interface AuthState { - /** The judgment backend this state describes; each backend has its own key. */ - readonly backend: TypeSafeBackend; + /** The judgment backend this state describes; each backend has its own key. Holds the value the caller passed. */ + readonly backend: BackendSpec; /** Same kinds as KeySituation: where the key in effect comes from. */ readonly kind: "environment" | "stored" | "missing" | "unusable"; readonly source?: KeySource; @@ -89,8 +89,12 @@ function writeState(path: string, state: { verifiedAt?: string; lastFailure?: Au } } -/** What the key situation, the last outcome, and the clock add up to for one backend. Never throws. */ -export function authState(options: { path?: string; backend?: TypeSafeBackend } = {}): AuthState { +/** + * What the key situation, the last outcome, and the clock add up to for a valid backend; it never throws for one. An + * invalid backend throws the same `configuration` error as resolveBackend(), so validate a user-supplied endpoint + * with resolveBackend() first. + */ +export function authState(options: { path?: string; backend?: BackendSpec } = {}): AuthState { const path = options.path ?? authStatePath(); const backend = options.backend ?? DEFAULT_BACKEND; const situation = keySituation(backend); @@ -146,7 +150,7 @@ export interface AuthReport { * state instead of reporting "enabled". */ export function describeAuth(state: AuthState = authState()): AuthReport { - const config = backendConfig(state.backend ?? DEFAULT_BACKEND); + const config = resolveBackend(state.backend); const label = `${config.label} key`; const since = state.lastFailure ? ` Last failure: ${state.lastFailure.message}${state.lastFailure.at ? ` (${state.lastFailure.at})` : ""}` : ""; if (state.kind === "missing") { diff --git a/src/backends.ts b/src/backends.ts index 0e8826a..28c1755 100644 --- a/src/backends.ts +++ b/src/backends.ts @@ -1,9 +1,9 @@ import { TypeSafeIntegrationError } from "./errors.js"; -export type TypeSafeBackend = "typesafe" | "openrouter"; +export type TypeSafeBackend = "typesafe" | "openrouter" | "commandcode"; export interface BackendConfig { - /** Human name for status lines: "TypeSafe", "OpenRouter". */ + /** Human name for status lines: "TypeSafe", "OpenRouter", "Command Code". */ label: string; host: string; /** The environment variable that carries this backend's key. Absent means the TypeSafe key resolution applies. */ @@ -20,13 +20,37 @@ export interface BackendConfig { modelsVerifyKey?: boolean; } +/** A caller-supplied endpoint that serves the Jev decisions protocol. Passed per call; never added to the registry. */ +export interface BackendEndpoint extends BackendConfig { + /** Required. The environment variable with this endpoint's key. Must not be TYPESAFE_API_KEY. */ + readonly keyEnv: string; + /** Model sent when the caller names none. Without it, `model` must be passed to createTypeSafe. */ + readonly defaultModel?: string; +} + +/** A registry name or a caller-supplied endpoint. */ +export type BackendSpec = TypeSafeBackend | BackendEndpoint; + +/** A backend resolved and validated: what the client will actually use. */ +export interface ResolvedBackend extends BackendConfig { + /** Registry name, absent for a caller-supplied endpoint. */ + readonly name?: TypeSafeBackend; + /** Origin only: scheme, host, and port. */ + readonly host: string; + readonly keyEnv: string; + /** The model id sent when the caller names none, already in the backend's form. Absent when the endpoint names none. */ + readonly defaultModel?: string; + /** Always explicit after resolution. */ + readonly modelsVerifyKey: boolean; +} + /** The backend every key and auth function assumes when none is named. */ export const DEFAULT_BACKEND: TypeSafeBackend = "typesafe"; /** The environment variable and login store that the default backend reads. */ export const TYPESAFE_KEY_ENV = "TYPESAFE_API_KEY"; -/** Registry of known judgment backends. Extendable by callers. */ +/** Registry of known judgment backends. Callers pass a `BackendEndpoint` for a host this registry does not name. */ export const DECISIONS_BACKENDS: Record = { typesafe: { label: "TypeSafe", host: "https://api.typesafe.ai", keyEnv: TYPESAFE_KEY_ENV }, openrouter: { @@ -39,6 +63,16 @@ export const DECISIONS_BACKENDS: Record = { modelsIdField: "id", modelsVerifyKey: false, }, + commandcode: { + label: "Command Code", + host: "https://api.commandcode.ai", + keyEnv: "COMMANDCODE_API_KEY", + path: "/provider/v1/systemone", + modelsPath: "/provider/v1/models", + modelsField: "data", + modelsIdField: "id", + modelsVerifyKey: false, + }, }; /** The registry entry for a backend name; a `configuration` error for a name the registry does not know. */ @@ -52,6 +86,7 @@ export function backendConfig(name: TypeSafeBackend): BackendConfig { const DEFAULT_MODEL: Record = { typesafe: "jev-latest", openrouter: "typesafe/jev-1.13", + commandcode: "typesafe/jev", }; /** @@ -81,3 +116,81 @@ export function defaultModelId(backend: TypeSafeBackend): string { export function usesTypesafeKey(backend: BackendConfig): boolean { return (backend.keyEnv ?? TYPESAFE_KEY_ENV) === TYPESAFE_KEY_ENV; } + +const KEY_ENV_MESSAGE = "Backend keyEnv must name an environment variable: letters, digits, and underscores, not starting with a digit."; +const HOST_MESSAGE = "Backend host must be an absolute https: URL with no user info, path, query, or fragment (http: is allowed only for localhost, 127.0.0.0/8, and [::1])."; + +function refuse(message: string): never { + throw new TypeSafeIntegrationError("configuration", message); +} + +/** `https://gw.example.com?` and `https://gw.example.com#` parse clean, so the raw string itself must carry neither. */ +function hasRawQueryOrFragment(value: string): boolean { + return value.includes("?") || value.includes("#"); +} + +/** + * Resolve a registry name or a caller-supplied endpoint into the validated form the client uses. A name resolves to + * its registry entry unchanged; an endpoint is validated field by field and returned as a fresh object whose `host` + * is origin-only. Messages never quote a caller value — a host can carry credentials in its user info — except the + * label, and only after it is validated. Validation runs on every call; nothing is cached. + */ +export function resolveBackend(backend?: BackendSpec): ResolvedBackend { + const spec: BackendSpec = backend === undefined ? DEFAULT_BACKEND : backend; + if (typeof spec === "string") { + const entry = backendConfig(spec); + return { + ...entry, + name: spec, + host: entry.host, + keyEnv: entry.keyEnv ?? TYPESAFE_KEY_ENV, + defaultModel: defaultModelId(spec), + modelsVerifyKey: entry.modelsVerifyKey !== false, + }; + } + if (spec === null || typeof spec !== "object") { + refuse("backend must be a registry name or a backend object."); + } + const label = typeof spec.label === "string" ? spec.label.trim() : ""; + if (!label || label.length > 60) refuse("Backend label must be a nonempty string of at most 60 characters."); + let host: URL; + if (typeof spec.host !== "string" || hasRawQueryOrFragment(spec.host)) refuse(HOST_MESSAGE); + try { + host = new URL(spec.host); + } catch { + refuse(HOST_MESSAGE); + } + const loopback = host.hostname === "localhost" || host.hostname === "[::1]" || /^127(?:\.\d{1,3}){3}$/.test(host.hostname); + if (host.protocol !== "https:" && !(host.protocol === "http:" && loopback)) refuse(HOST_MESSAGE); + if (host.username !== "" || host.password !== "" || host.pathname !== "/" || host.search !== "" || host.hash !== "") refuse(HOST_MESSAGE); + for (const [field, message] of [["path", 'Backend path must be a string that starts with "/".'], ["modelsPath", 'Backend modelsPath must be a string that starts with "/".']] as const) { + const value = spec[field]; + if (value !== undefined && (typeof value !== "string" || !value.startsWith("/") || hasRawQueryOrFragment(value))) refuse(message); + } + if (spec.modelsField !== undefined && (typeof spec.modelsField !== "string" || !spec.modelsField)) refuse("Backend modelsField must be a nonempty string."); + if (spec.modelsIdField !== undefined && (typeof spec.modelsIdField !== "string" || !spec.modelsIdField)) refuse("Backend modelsIdField must be a nonempty string."); + if (spec.modelsVerifyKey !== undefined && typeof spec.modelsVerifyKey !== "boolean") refuse("Backend modelsVerifyKey must be a boolean."); + if (typeof spec.keyEnv !== "string" || !/^[A-Za-z_][A-Za-z0-9_]*$/.test(spec.keyEnv)) refuse(KEY_ENV_MESSAGE); + if (spec.keyEnv.toUpperCase() === TYPESAFE_KEY_ENV) { + refuse("Backend keyEnv must not be TYPESAFE_API_KEY: the TypeSafe key is only sent to the typesafe backend. Give this endpoint its own variable."); + } + if (spec.defaultModel !== undefined && (typeof spec.defaultModel !== "string" || !spec.defaultModel.trim() || spec.defaultModel.length > 100)) { + refuse("Backend defaultModel must be a nonempty string of at most 100 characters."); + } + return { + label, + host: host.origin, + keyEnv: spec.keyEnv, + modelsVerifyKey: spec.modelsVerifyKey === true, + ...(spec.path === undefined ? {} : { path: spec.path }), + ...(spec.modelsPath === undefined ? {} : { modelsPath: spec.modelsPath }), + ...(spec.modelsField === undefined ? {} : { modelsField: spec.modelsField }), + ...(spec.modelsIdField === undefined ? {} : { modelsIdField: spec.modelsIdField }), + ...(spec.defaultModel === undefined ? {} : { defaultModel: spec.defaultModel }), + }; +} + +/** The destination host only — scheme, host, and port, e.g. `api.commandcode.ai` — for consent text. */ +export function backendHost(backend?: BackendSpec): string { + return new URL(resolveBackend(backend).host).host; +} diff --git a/src/client.ts b/src/client.ts index 56ee82e..20df7ad 100644 --- a/src/client.ts +++ b/src/client.ts @@ -1,8 +1,8 @@ import { TypeSafeClient } from "@typesafe-ai/sdk"; import type { Fetch, Questions, SystemOneRequest, SystemOneResult } from "@typesafe-ai/sdk"; import { recordAuthFailure, recordAuthVerified } from "./auth.js"; -import { DEFAULT_BACKEND, TYPESAFE_KEY_ENV, backendConfig, backendModelId, defaultModelId, usesTypesafeKey } from "./backends.js"; -import type { BackendConfig, TypeSafeBackend } from "./backends.js"; +import { TYPESAFE_KEY_ENV, backendModelId, resolveBackend, usesTypesafeKey } from "./backends.js"; +import type { BackendConfig, BackendSpec, ResolvedBackend } from "./backends.js"; import type { BatchEvaluation, BatchOptions } from "./batch.js"; import { evaluateAll, evaluateMany } from "./batch.js"; import { keySituation } from "./credentials.js"; @@ -11,8 +11,8 @@ import { DEFAULT_MAX_INPUT_BYTES, assertWithinByteLimit, prepareEvaluationReques import { DEFAULT_USD_PER_MTOK, capsFromEnvironment, estimateUsd, mergeCaps, openUsageLedger } from "./usage.js"; import type { BlockedCap, SpendCaps, UsageLedger, UsageReport } from "./usage.js"; -export { DECISIONS_BACKENDS, DEFAULT_BACKEND } from "./backends.js"; -export type { BackendConfig, TypeSafeBackend } from "./backends.js"; +export { backendHost, resolveBackend, DECISIONS_BACKENDS, DEFAULT_BACKEND } from "./backends.js"; +export type { BackendConfig, BackendEndpoint, BackendSpec, ResolvedBackend, TypeSafeBackend } from "./backends.js"; /** The paths the SDK appends to whatever base URL it is given. */ const SDK_PATH = "/v1/systemone"; @@ -61,8 +61,8 @@ async function translateModels(response: Response, backend: BackendConfig): Prom export interface TypeSafeOptions { /** Defaults to TYPESAFE_API_KEY, then the key saved by `/typesafe login`; never returned. */ apiKey?: string; - /** Judgment backend. When omitted, routes to the default TypeSafe host. */ - backend?: TypeSafeBackend; + /** Judgment backend: a registry name or a caller-supplied endpoint. When omitted, routes to the default TypeSafe host. */ + backend?: BackendSpec; /** Defaults to the backend's own default (`jev-latest`, `typesafe/jev-1.13` on OpenRouter); a bare Jev id is mapped to the backend's id form before sending. No model is inferred from submitted content. */ model?: string; /** Per request. Default: 15 seconds. No automatic retries. */ @@ -180,14 +180,15 @@ function capsDescription(caps: SpendCaps): string { /** A bounded, server-side TypeSafe client independent of Pi's runtime. */ export function createTypeSafe(options: TypeSafeOptions = {}): TypeSafe { + const backend: ResolvedBackend = resolveBackend(options.backend); let apiKey = options.apiKey?.trim(); - const backendName: TypeSafeBackend = options.backend ?? DEFAULT_BACKEND; - const backend = backendConfig(backendName); const baseURL = backend.host; + // Only a registry backend maps ids; a caller-supplied endpoint's model is sent as the caller wrote it. + const mapModel = (model: string): string => (backend.name === undefined ? model : backendModelId(backend.name, model)); if (!apiKey) { // The same resolution that authState() and ensureApiKey() report, so the status line and the request agree. - const situation = keySituation(backendName); + const situation = keySituation(options.backend); if (situation.kind === "unusable") throw new TypeSafeIntegrationError("configuration", situation.reason); if (situation.kind === "environment" || situation.kind === "stored") apiKey = situation.key; } @@ -209,9 +210,10 @@ export function createTypeSafe(options: TypeSafeOptions = {}): TypeSafe { const transport = backend.path !== undefined || backend.modelsPath !== undefined ? backendFetch(backend, options.fetch) : options.fetch; // The caller's input is validated as written, then mapped to the backend's id form; omitting it sends the backend's // own default, which the mapping leaves unchanged. - const requested = options.model ?? defaultModelId(backendName); + const requested = options.model ?? backend.defaultModel; + if (requested === undefined) throw new TypeSafeIntegrationError("configuration", `Backend "${backend.label}" names no defaultModel; pass model to createTypeSafe.`); if (typeof requested !== "string" || !requested.trim() || requested.length > 100) throw new TypeSafeIntegrationError("configuration", "model must be a nonempty string of at most 100 characters."); - const model = backendModelId(backendName, requested); + const model = mapModel(requested); // Do not inherit SDK debug logging or alternate destinations from the environment. const client = new TypeSafeClient({ apiKey, @@ -248,7 +250,7 @@ export function createTypeSafe(options: TypeSafeOptions = {}): TypeSafe { const models = await client.models.list(callOptions); if (!Array.isArray(models)) throw new TypeSafeIntegrationError("response", "TypeSafe returned an unexpected model list."); // A backend that serves its list publicly accepts any key, so a success there proves nothing about one. - if (backend.modelsVerifyKey !== false && !verificationRecorded) { + if (backend.modelsVerifyKey && !verificationRecorded) { verificationRecorded = true; recordAuthVerified(); } @@ -260,7 +262,7 @@ export function createTypeSafe(options: TypeSafeOptions = {}): TypeSafe { async evaluate(input: SystemOneRequest, callOptions: EvaluationOptions = {}): Promise> { const validated = prepareEvaluationRequest(input, { maxInputBytes }); // A per-request model meets the same mapping as the client default; the schema already limited the caller's own id. - const body = JSON.stringify({ ...validated, model: validated.model === undefined ? model : backendModelId(backendName, validated.model) }); + const body = JSON.stringify({ ...validated, model: validated.model === undefined ? model : mapModel(validated.model) }); assertWithinByteLimit(body, maxInputBytes); if (callOptions.signal?.aborted) throw new TypeSafeIntegrationError("aborted", "TypeSafe request cancelled before submission."); if (usage.requestsStarted >= maxRequests) { diff --git a/src/credentials.ts b/src/credentials.ts index e8a379a..d5215c2 100644 --- a/src/credentials.ts +++ b/src/credentials.ts @@ -1,13 +1,16 @@ import { chmodSync, mkdirSync, readFileSync, renameSync, rmSync, statSync, writeFileSync } from "node:fs"; import { homedir } from "node:os"; import { dirname, join } from "node:path"; -import { DEFAULT_BACKEND, TYPESAFE_KEY_ENV, backendConfig, usesTypesafeKey } from "./backends.js"; -import type { TypeSafeBackend } from "./backends.js"; +import { DEFAULT_BACKEND, TYPESAFE_KEY_ENV, resolveBackend, usesTypesafeKey } from "./backends.js"; +import type { BackendSpec } from "./backends.js"; import { TypeSafeIntegrationError } from "./errors.js"; export type KeySource = "environment" | "stored"; -/** The complete, never-throwing answer to "which key is in effect". */ +/** + * The complete answer to "which key is in effect" for a valid backend: it never throws. An invalid backend — an + * unknown name or an endpoint object that fails validation — throws the same `configuration` error as resolveBackend. + */ export type KeySituation = /** `keyEnv` names the variable that was read; absent means `TYPESAFE_API_KEY`. */ | { readonly kind: "environment"; readonly key: string; readonly keyEnv?: string } @@ -59,14 +62,16 @@ export function readStoredApiKey(): string | undefined { } /** - * What the environment, the login store, and file permissions add up to right now for one judgment backend. Never - * throws; the "unusable" kind carries the user-facing reason (a stored key that other local users can read). + * What the environment, the login store, and file permissions add up to right now for one judgment backend. A valid + * backend never throws: the "unusable" kind carries the user-facing reason (a stored key that other local users can + * read). An invalid backend — an unknown name or an endpoint object that fails validation — throws the same + * `configuration` error as resolveBackend(), so validate a user-supplied endpoint with resolveBackend() first. * The TypeSafe backend reads `TYPESAFE_API_KEY`, then the login store. Every other backend reads only its own * environment variable, because the store holds a TypeSafe key and a login verifies against api.typesafe.ai. */ -export function keySituation(backend: TypeSafeBackend = DEFAULT_BACKEND): KeySituation { - const config = backendConfig(backend); - const keyEnv = config.keyEnv ?? TYPESAFE_KEY_ENV; +export function keySituation(backend: BackendSpec = DEFAULT_BACKEND): KeySituation { + const config = resolveBackend(backend); + const keyEnv = config.keyEnv; const fromEnvironment = process.env[keyEnv]?.trim(); if (fromEnvironment) return { kind: "environment", key: fromEnvironment, keyEnv }; if (!usesTypesafeKey(config)) return { kind: "missing" }; @@ -94,10 +99,11 @@ export function keySourceLabel(situation: KeySituation): string { /** * The pre-0.4.0 key interface, frozen for existing callers: environment first so CI and scripts stay explicit, the * stored key as the interactive default, `undefined` when no key is configured, and a `configuration` error when a - * store must not be read. New code should call keySituation() instead: same precedence, never throws, and the - * "must not be read" case arrives as `unusable` with the reason. + * store must not be read. New code should call keySituation() instead: same precedence, never throws for a valid + * backend (an invalid one throws resolveBackend's `configuration` error), and the "must not be read" case arrives as + * `unusable` with the reason. */ -export function resolveApiKey(backend: TypeSafeBackend = DEFAULT_BACKEND): { key: string; source: KeySource } | undefined { +export function resolveApiKey(backend: BackendSpec = DEFAULT_BACKEND): { key: string; source: KeySource } | undefined { const situation = keySituation(backend); if (situation.kind === "unusable") throw new TypeSafeIntegrationError("configuration", situation.reason); return situation.kind === "environment" || situation.kind === "stored" ? { key: situation.key, source: situation.kind } : undefined; diff --git a/src/errors.ts b/src/errors.ts index 9d480be..194da57 100644 --- a/src/errors.ts +++ b/src/errors.ts @@ -1,6 +1,6 @@ import { APIError, APIConnectionError, APITimeoutError, APIUserAbortError } from "@typesafe-ai/sdk"; import { DECISIONS_BACKENDS, TYPESAFE_KEY_ENV, backendConfig } from "./backends.js"; -import type { BackendConfig, TypeSafeBackend } from "./backends.js"; +import type { BackendConfig, BackendSpec } from "./backends.js"; export type IntegrationErrorCode = "configuration" | "validation" | "budget" | "aborted" | "timeout" | "http" | "connection" | "response"; @@ -31,7 +31,7 @@ function retryAfterSeconds(error: APIError): number | undefined { * Classify an error into a message safe to display. `backend` names the key variable the 401 advice tells the user * to check and selects the 402 wording; omitting it keeps the one-argument call and assumes the default TypeSafe key. */ -export function safeError(error: unknown, backend?: TypeSafeBackend | BackendConfig): TypeSafeIntegrationError { +export function safeError(error: unknown, backend?: BackendSpec | BackendConfig): TypeSafeIntegrationError { if (error instanceof TypeSafeIntegrationError) return error; if (error instanceof APIUserAbortError) return new TypeSafeIntegrationError("aborted", "TypeSafe request cancelled; an already submitted request may still be billed."); if (error instanceof APITimeoutError) return new TypeSafeIntegrationError("timeout", "TypeSafe request timed out; it was not retried and may still be billed."); diff --git a/src/index.ts b/src/index.ts index b362610..6f1f0b5 100644 --- a/src/index.ts +++ b/src/index.ts @@ -1,7 +1,7 @@ -export { createTypeSafe, DEFAULT_MAX_REQUESTS, DECISIONS_BACKENDS, DEFAULT_BACKEND } from "./client.js"; +export { createTypeSafe, backendHost, resolveBackend, DEFAULT_MAX_REQUESTS, DECISIONS_BACKENDS, DEFAULT_BACKEND } from "./client.js"; export type { TypeSafe, TypeSafeOptions, EvaluationOptions, Evaluation, UsageSnapshot, SpendReport, - TypeSafeBackend, BackendConfig, + TypeSafeBackend, BackendConfig, BackendEndpoint, BackendSpec, ResolvedBackend, } from "./client.js"; export { ask, DEFAULT_ASK_TIMEOUT_MS } from "./ask.js"; export type { AskAnswer, AskOptions, Judge } from "./ask.js"; diff --git a/src/login.ts b/src/login.ts index 66b99ce..92438fb 100644 --- a/src/login.ts +++ b/src/login.ts @@ -1,6 +1,6 @@ import type { ExtensionCommandContext } from "@earendil-works/pi-coding-agent"; -import { DEFAULT_BACKEND, backendConfig, usesTypesafeKey } from "./backends.js"; -import type { TypeSafeBackend } from "./backends.js"; +import { DEFAULT_BACKEND, resolveBackend, usesTypesafeKey } from "./backends.js"; +import type { BackendSpec } from "./backends.js"; import { createTypeSafe } from "./client.js"; import { keySituation, normalizeApiKey, storeApiKey } from "./credentials.js"; import type { KeySource } from "./credentials.js"; @@ -46,12 +46,12 @@ export type EnsureApiKeyResult = * (every backend except TypeSafe) throws `configuration` naming its environment variable instead of prompting, because * the prompt would verify the key against the wrong service and store it where the TypeSafe key lives. */ -export async function ensureApiKey(ctx: ExtensionCommandContext, options: { backend?: TypeSafeBackend } = {}): Promise { +export async function ensureApiKey(ctx: ExtensionCommandContext, options: { backend?: BackendSpec } = {}): Promise { const backend = options.backend ?? DEFAULT_BACKEND; const situation = keySituation(backend); if (situation.kind === "environment" || situation.kind === "stored") return { source: situation.kind }; if (situation.kind === "unusable") throw new TypeSafeIntegrationError("configuration", situation.reason); - const config = backendConfig(backend); + const config = resolveBackend(backend); if (!usesTypesafeKey(config)) { throw new TypeSafeIntegrationError("configuration", `No ${config.label} key. Set ${config.keyEnv} in the environment; /typesafe login stores a TypeSafe key only.`); } diff --git a/tests/backends.test.ts b/tests/backends.test.ts new file mode 100644 index 0000000..5e2cb6d --- /dev/null +++ b/tests/backends.test.ts @@ -0,0 +1,380 @@ +import assert from "node:assert/strict"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { after, before, beforeEach, test } from "node:test"; +import { + authState, backendHost, clearAuthState, clearStoredApiKey, createTypeSafe, keySituation, + noul, resolveBackend, storeApiKey, TypeSafeIntegrationError, +} from "../src/index.js"; +import type { BackendEndpoint, BackendSpec, Questions } from "../src/index.js"; +import { safeError } from "../src/errors.js"; +import { APIError } from "@typesafe-ai/sdk"; + +function responseFor(questions: Questions): Response { + const answers = Object.fromEntries(Object.entries(questions).map(([id, q]) => { + if (q.type === "noul") return [id, { type: "noul", noul: 0.9 }]; + if (q.type === "choice") { + const keys = Object.keys(q.criteria); + return [id, { type: "choice", choice: keys[0], confidence: 1, probabilities: Object.fromEntries(keys.map((key, i) => [key, i === 0 ? 1 : 0])) }]; + } + return [id, { type: "score", score: 0, confidence: 1, probabilities: Object.fromEntries(q.criteria.map((_, i) => [i, i === 0 ? 1 : 0])), legend: Object.fromEntries(q.criteria.map((level, i) => [i, level])) }]; + })); + return Response.json({ model: "jev-test", answers, usage: { input_tokens: 42, output_tokens: 0 } }); +} + +const sample = () => ({ state: "synthetic", questions: { yes: noul("Is this synthetic?") } }); +const gateway: BackendEndpoint = { label: "Gateway", host: "https://gw.example.com", path: "/jev/v1/systemone", keyEnv: "GATEWAY_JEV_KEY", defaultModel: "jev-latest" }; +// Obviously fake keys only; nothing here ever reaches a network. +const ccKey = "cc_fake_key_0123456789abcdef"; +const gwKey = "gw_fake_key_0123456789abcdef"; +const tsKey = "ts_fake_key_0123456789abcdef"; +const storedKey = "st_fake_key_0123456789abcdef"; + +const savedAgentDir = process.env.PI_CODING_AGENT_DIR; +const savedEnv = new Map(["TYPESAFE_API_KEY", "COMMANDCODE_API_KEY", "GATEWAY_JEV_KEY"].map(name => [name, process.env[name]])); +before(() => { + process.env.PI_CODING_AGENT_DIR = mkdtempSync(join(tmpdir(), "pi-typesafe-backends-")); +}); +beforeEach(() => { + for (const name of ["TYPESAFE_API_KEY", "COMMANDCODE_API_KEY", "GATEWAY_JEV_KEY"]) delete process.env[name]; + clearStoredApiKey(); + clearAuthState(); +}); +after(() => { + if (savedAgentDir === undefined) delete process.env.PI_CODING_AGENT_DIR; else process.env.PI_CODING_AGENT_DIR = savedAgentDir; + for (const [name, value] of savedEnv) { + if (value === undefined) delete process.env[name]; else process.env[name] = value; + } +}); + +function assertRefuses(backend: unknown, message: string, options: { apiKey?: string; model?: string } = {}): void { + let fetchCalls = 0; + const fetch = async () => { fetchCalls += 1; return new Response("{}"); }; + assert.throws(() => createTypeSafe({ + backend: backend as BackendSpec, + fetch, + ...(options.apiKey === undefined ? {} : { apiKey: options.apiKey }), + ...(options.model === undefined ? {} : { model: options.model }), + }), (error: unknown) => { + assert.ok(error instanceof TypeSafeIntegrationError); + assert.equal(error.code, "configuration"); + assert.equal(error.message, message); + assert.ok(!error.message.includes("user:pw")); + return true; + }); + assert.equal(fetchCalls, 0); +} + +// 1. Command Code request. + +test("a commandcode request goes to the Command Code path with its own key and model", async () => { + process.env.COMMANDCODE_API_KEY = ccKey; + const questions = sample().questions; + let calls = 0; + const client = createTypeSafe({ backend: "commandcode", fetch: async (url, init) => { + calls += 1; + assert.equal(String(url), "https://api.commandcode.ai/provider/v1/systemone"); + assert.equal(new Headers(init?.headers).get("authorization"), `Bearer ${ccKey}`); + assert.equal((JSON.parse(String(init?.body)) as { model: string }).model, "typesafe/jev"); + return responseFor(questions); + } }); + await client.evaluate(sample()); + assert.equal(calls, 1); +}); + +test("a per-request model on commandcode is sent unchanged", async () => { + process.env.COMMANDCODE_API_KEY = ccKey; + const questions = sample().questions; + const client = createTypeSafe({ backend: "commandcode", fetch: async (_url, init) => { + assert.equal((JSON.parse(String(init?.body)) as { model: string }).model, "jev-2.0"); + return responseFor(questions); + } }); + await client.evaluate({ ...sample(), model: "jev-2.0" }); +}); + +// 2. Custom object. + +test("a caller-supplied endpoint sends to its own path with its own key and unmapped model", async () => { + process.env.GATEWAY_JEV_KEY = gwKey; + const questions = sample().questions; + let calls = 0; + const client = createTypeSafe({ backend: gateway, fetch: async (url, init) => { + calls += 1; + assert.equal(String(url), "https://gw.example.com/jev/v1/systemone"); + assert.equal(new Headers(init?.headers).get("authorization"), `Bearer ${gwKey}`); + assert.equal((JSON.parse(String(init?.body)) as { model: string }).model, "jev-latest"); + return responseFor(questions); + } }); + await client.evaluate(sample()); + assert.equal(calls, 1); +}); + +test("an endpoint without a path uses the SDK's own /v1/systemone", async () => { + process.env.GATEWAY_JEV_KEY = gwKey; + const questions = sample().questions; + let sentUrl = ""; + const client = createTypeSafe({ backend: { label: "Gateway", host: "https://gw.example.com", keyEnv: "GATEWAY_JEV_KEY", defaultModel: "jev-latest" }, fetch: async (url) => { + sentUrl = String(url); + return responseFor(questions); + } }); + await client.evaluate(sample()); + assert.equal(sentUrl, "https://gw.example.com/v1/systemone"); +}); + +test("an endpoint may use a loopback http: host", async () => { + process.env.GATEWAY_JEV_KEY = gwKey; + const questions = sample().questions; + let sentUrl = ""; + const client = createTypeSafe({ backend: { label: "Local", host: "http://127.0.0.1:8787", keyEnv: "GATEWAY_JEV_KEY", defaultModel: "jev-latest" }, fetch: async (url) => { + sentUrl = String(url); + return responseFor(questions); + } }); + await client.evaluate(sample()); + assert.equal(sentUrl, "http://127.0.0.1:8787/v1/systemone"); +}); + +// 3. Every refusal in the validation table. + +const hostMessage = "Backend host must be an absolute https: URL with no user info, path, query, or fragment (http: is allowed only for localhost, 127.0.0.0/8, and [::1])."; + +test("an endpoint label must be a nonempty string of at most 60 characters", () => { + for (const label of [7, "", " ", "x".repeat(61)]) { + assertRefuses({ label, host: "https://gw.example.com", keyEnv: "GATEWAY_JEV_KEY", defaultModel: "jev-latest" }, "Backend label must be a nonempty string of at most 60 characters."); + } +}); + +test("an endpoint host must be an absolute https: URL with no user info, path, query, or fragment", () => { + for (const host of ["http://gw.example.com", "https://user:pw@gw.example.com", "https://gw.example.com/prefix", "https://gw.example.com?x=1", "https://gw.example.com#f", "ftp://gw.example.com", "not a url"]) { + assertRefuses({ label: "Gateway", host, keyEnv: "GATEWAY_JEV_KEY", defaultModel: "jev-latest" }, hostMessage); + } +}); + +test('an endpoint path must be a string that starts with "/"', () => { + for (const path of ["jev", "x?q", "x#f", 7]) { + assertRefuses({ label: "Gateway", host: "https://gw.example.com", keyEnv: "GATEWAY_JEV_KEY", defaultModel: "jev-latest", path }, 'Backend path must be a string that starts with "/".'); + } +}); + +test('an endpoint modelsPath must be a string that starts with "/"', () => { + for (const modelsPath of ["models", "x?q", 7]) { + assertRefuses({ label: "Gateway", host: "https://gw.example.com", keyEnv: "GATEWAY_JEV_KEY", defaultModel: "jev-latest", modelsPath }, 'Backend modelsPath must be a string that starts with "/".'); + } +}); + +test("an endpoint modelsField must be a nonempty string", () => { + for (const modelsField of ["", 7]) { + assertRefuses({ label: "Gateway", host: "https://gw.example.com", keyEnv: "GATEWAY_JEV_KEY", defaultModel: "jev-latest", modelsField }, "Backend modelsField must be a nonempty string."); + } +}); + +test("an endpoint modelsIdField must be a nonempty string", () => { + for (const modelsIdField of ["", 7]) { + assertRefuses({ label: "Gateway", host: "https://gw.example.com", keyEnv: "GATEWAY_JEV_KEY", defaultModel: "jev-latest", modelsIdField }, "Backend modelsIdField must be a nonempty string."); + } +}); + +test("an endpoint modelsVerifyKey must be a boolean", () => { + for (const modelsVerifyKey of ["yes", 1]) { + assertRefuses({ label: "Gateway", host: "https://gw.example.com", keyEnv: "GATEWAY_JEV_KEY", defaultModel: "jev-latest", modelsVerifyKey }, "Backend modelsVerifyKey must be a boolean."); + } +}); + +test("an endpoint keyEnv must name an environment variable", () => { + for (const keyEnv of ["1BAD", "GATEWAY KEY", ""]) { + assertRefuses({ label: "Gateway", host: "https://gw.example.com", keyEnv, defaultModel: "jev-latest" }, "Backend keyEnv must name an environment variable: letters, digits, and underscores, not starting with a digit."); + } +}); + +test("an endpoint keyEnv must not be TYPESAFE_API_KEY", () => { + const message = "Backend keyEnv must not be TYPESAFE_API_KEY: the TypeSafe key is only sent to the typesafe backend. Give this endpoint its own variable."; + for (const keyEnv of ["TYPESAFE_API_KEY", "typesafe_api_key"]) { + assertRefuses({ label: "Gateway", host: "https://gw.example.com", keyEnv, defaultModel: "jev-latest" }, message); + } +}); + +test("an endpoint defaultModel must be a nonempty string of at most 100 characters", () => { + for (const defaultModel of ["", " ", "x".repeat(101), 7]) { + assertRefuses({ label: "Gateway", host: "https://gw.example.com", keyEnv: "GATEWAY_JEV_KEY", defaultModel }, "Backend defaultModel must be a nonempty string of at most 100 characters."); + } +}); + +test("an endpoint with no defaultModel needs model on createTypeSafe", () => { + assertRefuses({ label: "Gateway", host: "https://gw.example.com", keyEnv: "GATEWAY_JEV_KEY" }, 'Backend "Gateway" names no defaultModel; pass model to createTypeSafe.', { apiKey: "fake_key_0123456789abcdef" }); +}); + +test("backend must be a registry name or a backend object", () => { + for (const backend of [42, null, true]) { + assertRefuses(backend, "backend must be a registry name or a backend object."); + } + // An unknown name keeps the registry's own error. + assertRefuses("unknown-backend", 'Unknown judgment backend "unknown-backend". Valid backends: typesafe, openrouter, commandcode.'); +}); + +test("authState refuses an invalid backend instead of reporting a status", () => { + assert.throws(() => authState({ backend: { label: "x", host: "http://evil.example", keyEnv: "K" } }), (error: unknown) => { + assert.ok(error instanceof TypeSafeIntegrationError); + assert.equal(error.code, "configuration"); + assert.equal(error.message, "Backend host must be an absolute https: URL with no user info, path, query, or fragment (http: is allowed only for localhost, 127.0.0.0/8, and [::1])."); + return true; + }); +}); + +// 4. Key isolation. + +test("an endpoint never reads TYPESAFE_API_KEY or the login store", () => { + process.env.TYPESAFE_API_KEY = tsKey; + storeApiKey(storedKey); + let fetchCalls = 0; + const fetch = async () => { fetchCalls += 1; return new Response("{}"); }; + assert.throws(() => createTypeSafe({ backend: { label: "Gateway", host: "https://gw.example.com", keyEnv: "GATEWAY_JEV_KEY", defaultModel: "jev-latest" }, fetch }), (error: unknown) => { + assert.ok(error instanceof TypeSafeIntegrationError); + assert.equal(error.code, "configuration"); + assert.equal(error.message, "No API key. Set GATEWAY_JEV_KEY in the environment."); + return true; + }); + assert.equal(fetchCalls, 0); + assert.equal(keySituation(gateway).kind, "missing"); + assert.equal(authState({ backend: gateway }).usable, false); +}); + +test("commandcode with no COMMANDCODE_API_KEY is missing and unusable", () => { + process.env.TYPESAFE_API_KEY = tsKey; + storeApiKey(storedKey); + let fetchCalls = 0; + const fetch = async () => { fetchCalls += 1; return new Response("{}"); }; + assert.throws(() => createTypeSafe({ backend: "commandcode", fetch }), (error: unknown) => { + assert.ok(error instanceof TypeSafeIntegrationError); + assert.equal(error.code, "configuration"); + assert.equal(error.message, "No API key. Set COMMANDCODE_API_KEY in the environment."); + return true; + }); + assert.equal(fetchCalls, 0); + assert.equal(keySituation("commandcode").kind, "missing"); + assert.equal(authState({ backend: "commandcode" }).usable, false); +}); + +test("a request to an endpoint carries its own key, never the TypeSafe key", async () => { + process.env.TYPESAFE_API_KEY = tsKey; + process.env.GATEWAY_JEV_KEY = gwKey; + storeApiKey(storedKey); + const questions = sample().questions; + let calls = 0; + const client = createTypeSafe({ backend: gateway, fetch: async (_url, init) => { + calls += 1; + const headers = new Headers(init?.headers); + for (const value of headers.values()) { + assert.ok(!value.includes(tsKey), "the TypeSafe key must never reach another backend"); + assert.ok(!value.includes(storedKey), "the stored login key must never reach another backend"); + } + assert.equal(headers.get("authorization"), `Bearer ${gwKey}`); + return responseFor(questions); + } }); + await client.evaluate(sample()); + assert.equal(calls, 1); +}); + +// 5. A model list does not verify the key. + +test("a public model list on commandcode leaves the auth state unverified", async () => { + process.env.COMMANDCODE_API_KEY = ccKey; + const client = createTypeSafe({ backend: "commandcode", fetch: async () => Response.json({ data: [{ id: "typesafe/jev" }] }) }); + assert.deepEqual(await client.listModels(), ["typesafe/jev"]); + const state = authState({ backend: "commandcode" }); + assert.equal(state.verified, false); + assert.equal(state.verifiedAt, undefined); +}); + +test("a model list on an endpoint without modelsVerifyKey leaves the auth state unverified", async () => { + process.env.GATEWAY_JEV_KEY = gwKey; + const client = createTypeSafe({ backend: { label: "Gateway", host: "https://gw.example.com", keyEnv: "GATEWAY_JEV_KEY", defaultModel: "jev-latest" }, fetch: async () => Response.json({ models: [{ name: "jev-latest" }] }) }); + assert.deepEqual(await client.listModels(), ["jev-latest"]); + const state = authState({ backend: gateway }); + assert.equal(state.verified, false); + assert.equal(state.verifiedAt, undefined); +}); + +test("an endpoint with modelsVerifyKey true records verification", async () => { + process.env.GATEWAY_JEV_KEY = gwKey; + const endpoint: BackendEndpoint = { label: "Gateway", host: "https://gw.example.com", keyEnv: "GATEWAY_JEV_KEY", defaultModel: "jev-latest", modelsVerifyKey: true }; + const client = createTypeSafe({ backend: endpoint, fetch: async () => Response.json({ models: [{ name: "jev-latest" }] }) }); + await client.listModels(); + const state = authState({ backend: endpoint }); + assert.equal(state.verified, true); + assert.ok(state.verifiedAt); +}); + +test("a successful evaluate records verification even after a public model list", async () => { + process.env.COMMANDCODE_API_KEY = ccKey; + const questions = sample().questions; + const client = createTypeSafe({ backend: "commandcode", fetch: async (url) => String(url).endsWith("/models") + ? Response.json({ data: [{ id: "typesafe/jev" }] }) + : responseFor(questions) }); + await client.listModels(); + assert.equal(authState({ backend: "commandcode" }).verified, false); + await client.evaluate(sample()); + assert.equal(authState({ backend: "commandcode" }).verified, true); +}); + +// 6. Malformed replies stay `response` errors. + +test("a malformed reply on commandcode is a response error", async () => { + process.env.COMMANDCODE_API_KEY = ccKey; + const client = createTypeSafe({ backend: "commandcode", fetch: async () => Response.json({ model: "jev-test", answers: {}, usage: { input_tokens: 1, output_tokens: 0 } }) }); + await assert.rejects(client.evaluate(sample()), (error: unknown) => { + assert.ok(error instanceof TypeSafeIntegrationError); + assert.equal(error.code, "response"); + return true; + }); +}); + +test("a malformed reply on an endpoint is a response error", async () => { + process.env.GATEWAY_JEV_KEY = gwKey; + const client = createTypeSafe({ backend: gateway, fetch: async () => Response.json({ model: "jev-test", answers: { yes: { type: "noul", noul: 2 } }, usage: { input_tokens: 1, output_tokens: 0 } }) }); + await assert.rejects(client.evaluate(sample()), (error: unknown) => { + assert.ok(error instanceof TypeSafeIntegrationError); + assert.equal(error.code, "response"); + return true; + }); +}); + +// 7. 401 advice names each backend's own key variable. + +test("401 advice names each backend's own key variable", () => { + assert.equal(safeError(new APIError(401, undefined, new Headers()), "commandcode").message, "TypeSafe returned HTTP 401. Check COMMANDCODE_API_KEY. No automatic retry was made."); + assert.equal(safeError(new APIError(401, undefined, new Headers()), gateway).message, "TypeSafe returned HTTP 401. Check GATEWAY_JEV_KEY. No automatic retry was made."); +}); + +// 8. resolveBackend and backendHost. + +test("resolveBackend resolves registry names to the registry entries", () => { + const typesafe = resolveBackend("typesafe"); + assert.equal(typesafe.name, "typesafe"); + assert.equal(typesafe.label, "TypeSafe"); + assert.equal(typesafe.host, "https://api.typesafe.ai"); + assert.equal(typesafe.keyEnv, "TYPESAFE_API_KEY"); + assert.equal(typesafe.defaultModel, "jev-latest"); + assert.equal(typesafe.modelsVerifyKey, true); + assert.equal(typesafe.path, undefined); + const commandcode = resolveBackend("commandcode"); + assert.equal(commandcode.name, "commandcode"); + assert.equal(commandcode.label, "Command Code"); + assert.equal(commandcode.host, "https://api.commandcode.ai"); + assert.equal(commandcode.keyEnv, "COMMANDCODE_API_KEY"); + assert.equal(commandcode.path, "/provider/v1/systemone"); + assert.equal(commandcode.modelsPath, "/provider/v1/models"); + assert.equal(commandcode.modelsField, "data"); + assert.equal(commandcode.modelsIdField, "id"); + assert.equal(commandcode.defaultModel, "typesafe/jev"); + assert.equal(commandcode.modelsVerifyKey, false); + assert.deepEqual(resolveBackend("openrouter").defaultModel, "typesafe/jev-1.13"); +}); + +test("backendHost reports the destination host", () => { + assert.equal(backendHost("commandcode"), "api.commandcode.ai"); + assert.equal(backendHost("typesafe"), "api.typesafe.ai"); + assert.equal(backendHost("openrouter"), "openrouter.ai"); + assert.equal(backendHost(gateway), "gw.example.com"); + assert.equal(backendHost({ label: "Gateway", host: "https://gw.example.com:8443", keyEnv: "GATEWAY_JEV_KEY" }), "gw.example.com:8443"); + assert.equal(backendHost({ label: "Local", host: "http://127.0.0.1:8787", keyEnv: "GATEWAY_JEV_KEY" }), "127.0.0.1:8787"); +});