Skip to content

fix: ask each backend for its own model list - #12

Merged
DevMortimer merged 3 commits into
DevMortimer:mainfrom
SamuelMauricioL:fix/models-endpoint-per-backend
Sep 23, 2026
Merged

DevMortimer merged 3 commits into
DevMortimer:mainfrom
SamuelMauricioL:fix/models-endpoint-per-backend

Conversation

@SamuelMauricioL

Copy link
Copy Markdown
Contributor

What and why

Closes #11.

listModels() asked every backend for the SDK's own /v1/models, because the transport wrapper only rewrote the judgment path. An OpenRouter client therefore requested https://openrouter.ai/v1/models, which answers with the app's HTML page, and the SDK rejected the body. So listModels() could never verify a key on that backend — the documented way to prove one without spending the request budget.

Change

  • BackendConfig gains modelsPath? and modelsField?; DECISIONS_BACKENDS.openrouter declares /api/v1/models and data.
  • The transport wrapper rewrites the model-list path as well as the judgment path, and renames the list to the models field the SDK reads. A body without the declared field is passed through unchanged, so the SDK still reports its own shape error rather than a masked one.
  • The SDK is untouched. The TypeSafe backend declares neither field, installs no wrapper, and keeps the caller's fetch exactly as before.
  • docs/api.md documents the two fields.

Verification

  • npm ci and npm run check pass: build, typecheck, 99 offline tests, three of them new.
  • The new tests run offline through the injected fetch: the URL each backend is asked for, the renamed envelope, the pass-through when the declared field is absent, and that a TypeSafe response is never renamed.
  • docs/api.md updated.

Live testing: this touches the transport, so I checked the changed path against the real endpoint. With backend: "openrouter", listModels() returns 454 models from https://openrouter.ai/api/v1/models. That endpoint is public, so the check sent no key. I did not run npm run test:live: it bills a request against a TypeSafe key and I do not have one.

Compatibility and release notes

  • Additive: two optional fields on BackendConfig, both documented. No export added or removed, and no behavior change for backend: "typesafe" or for a caller that omits backend.
  • Known limitation, left alone here as a separate topic: OpenRouter's list carries display names (Xiaomi: MiMo-V2.6-Pro-UltraSpeed), not the model ids that model: accepts. Every entry carries a name and the longest is 56 characters, so the existing 100-character filter drops none.
  • No version bump.

listModels() requested the SDK's own /v1/models against the backend's
host. Only the judgment path was rewritten, so an OpenRouter client
asked for a path that host does not serve: openrouter.ai answers it with
its app page and the SDK rejects the body. listModels() could never
verify a key on that backend, which is the budget-free way to prove one.

The transport wrapper now rewrites the model-list path too, and renames
the list to the field the SDK reads when the registry declares another.
Both are registry data, so the SDK stays untouched and the TypeSafe
backend keeps the caller's fetch exactly as before.

Closes DevMortimer#11

@DevMortimer DevMortimer left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the path rewrite and field rename look right, and the TypeSafe backend is untouched. Two things to fix in this branch before merge:

  1. A bad OpenRouter key now gets recorded as verified. OpenRouter's /api/v1/models is public, so listModels() succeeds with any key, and the success path calls recordAuthVerified(). After this PR authState({ backend: "openrouter" }) reports verified: true for a missing or garbage key. Before, that path was unreachable for OpenRouter. Please skip the verified write when the backend's models endpoint does not check the key, for example an optional flag on BackendConfig, and narrow the "listModels() verifies the key" line in docs/api.md to match. An offline test that OpenRouter listModels() leaves the auth state unverified would cover it.

  2. Return model ids, not display names. Please handle the limitation you noted here rather than separately: OpenRouter entries carry id, which is what model: accepts, while name is a label. Have listModels() return usable ids for OpenRouter, and keep the TypeSafe backend's output unchanged.

Review feedback on the model-list change.

openrouter.ai serves /api/v1/models to anyone, so a success there says
nothing about the key. Recording it as verified would report a missing
or garbage OpenRouter key as good, which the unreachable path never did.
The registry now says whether a list checks the key, and only a list that
does writes the verification record.

OpenRouter entries carry the id that `model:` accepts and a display name,
so the entry's id is promoted to the name listModels() returns. The
TypeSafe backend declares neither and is unchanged.
@SamuelMauricioL

Copy link
Copy Markdown
Contributor Author

Both fixed in 94273b5.

1. A public list no longer records verification. BackendConfig gains modelsVerifyKey, absent meaning the list checks the key; OpenRouter sets it to false. listModels() now writes the verification record only when the list checks the key, and a backend that skips it leaves verificationRecorded alone, so a later successful evaluate() still records one. Checked against both hosts rather than assumed: GET https://openrouter.ai/api/v1/models answers 200 with no key and with a garbage key, while https://api.typesafe.ai/v1/models answers 401 for a garbage key. A new offline test asserts that OpenRouter listModels() leaves authState({ backend: "openrouter" }).verified false.

2. listModels() returns ids for OpenRouter. BackendConfig gains modelsIdField, and the transport promotes each entry's declared field to the name the SDK hands back, so the returned strings are what model: accepts. TypeSafe declares neither field, so its output is unchanged.

docs/api.md documents both fields and narrows the listModels() claim to exclude a backend whose list is public.

npm run check: 100 tests, 100 pass. Against the live endpoint: 454 entries, all ids (cohere/command-a-plus, openai/gpt-6-luna-pro, …), and the auth state stays unverified.

@DevMortimer DevMortimer left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both points addressed. Verified locally on 94273b5: build, typecheck, 100 tests pass; OpenRouter listModels() leaves the auth state unverified and returns ids.

@DevMortimer
DevMortimer merged commit 72e4b5b into DevMortimer:main Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

listModels cannot verify a key on a non-TypeSafe backend

2 participants