`--key` push signing has no install-time verification path (plugins)

[samuv]

Status update 2026-09-03. The skills half of this issue is resolved; the plugins half is not, and its shape changed. Original diagnosis preserved below, since it still explains why the failure cannot be overridden.

• Skills: done. Install-time key verification landed via Report key-signed artifacts as such at install #6443, Record a pinned cosign key in the lock schema #6444, Verify key-signed skill installs against a public key #6447 and Honour a pinned cosign key on sync and upgrade #6478 — skills.InstallOptions.PublicKey, lockfile.Provenance.PublicKey, install (pkg/skills/skillsvc/verify.go:264), upgrade (upgrade.go:283) and offline sync (sync.go:307) all verify against a supplied or pinned key. thv skill push --key now documents the --public-key step consumers need.
• Plugins: --key was removed rather than fixed. Sign plugin pushes and remove the lock feature gate #6438 shipped plugin push as keyless-only and omits Key from plugins.PushOptions entirely, so the flag cannot be constructed by any caller, in-process or over HTTP. That closed the publish-only dead end by removing the publish side — it did not give plugins the key path skills now has.

What remains is one decision, not the original bug: implement the skills key path for plugins, or ratify keyless-only for plugins permanently. #6300 recorded keyless-only as the agreed interim contract when it closed.

Remaining scope (plugins)

Plugins now have no key surface at all — no --key at push, no PublicKey on install options, no key branch in verify or sync:

Concern	Skills (done)	Plugins (absent)
--key push flag	cmd/thv/app/skill_push.go:35	none — cmd/thv/app/ai_plugin_push.go:28 documents why
Public key on install	skills.InstallOptions.PublicKey (pkg/skills/options.go:51)	none
Pinned key in lock	lockfile.Provenance.PublicKey (pkg/skills/lockfile/lockfile.go:140)	shared schema, unused by plugins
Install verify	pkg/skills/skillsvc/verify.go:264	keyless only (pkg/plugins/pluginsvc/verify.go)
Upgrade verify	pkg/skills/skillsvc/upgrade.go:283	keyless only
Offline sync re-verify	pkg/skills/skillsvc/sync.go:307	keyless only (pkg/plugins/pluginsvc/sync.go)

The lock schema already carries publicKey, so the plugins work is install options, the install API DTO, the verify/upgrade key branches, the sync re-verify branch, and finally restoring the push flag. Restoring --key first would recreate exactly the dead end this issue was opened about, so it has to come last.

Original diagnosis (still accurate for plugins)

thv skill push --key and thv ai-plugin push --key both accepted a cosign private key and produced a validly key-signed artifact — but install-time verification was keyless-only, so neither could be installed project-scoped. --key was a publish-only dead end on both surfaces.

It is worse than "installs as unsigned". Tracing the failure:

1. retrieveBundles does find the key-signed bundle — it is a real signature referrer — so ErrUnsigned never fires. That sentinel means no signature material at all (pkg/skills/verifier/errors.go).
2. Keyless verification then fails, because a key-pair bundle carries no Fulcio certificate chain to check against the keyless trusted material.
3. Classification finds no signer mismatch and no expected identity on first use, so it falls through to ErrSignatureInvalid.
4. isAllowedUnsigned requires errors.Is(err, verifier.ErrUnsigned) — false here — so --allow-unsigned does not override it.

Net effect: a hard 403 with no escape hatch. This is why the resolution could not be "let it install as unsigned", and why removing the plugin flag was preferred to leaving it in place with a warning.

Reproduce

Skills — now works, and is the shape plugins should acquire:

cosign generate-key-pair
thv skill push --key cosign.key ghcr.io/<org>/<skill>:v1
thv skill install ghcr.io/<org>/<skill>:v1 --project-root . --public-key <base64 DER SPKI>

Plugins — not reproducible: thv ai-plugin push has no --key to pass.

Expected behavior

Originally either of:

• Support it. A key-signed artifact can be installed project-scoped by supplying the matching public key, the pinned key is recorded in the lock entry, and offline sync --check re-verifies against it.
• Or reject it at push time, so users cannot publish an artifact ToolHive itself refuses to consume.

Skills took the first branch. Plugins took the second, by not offering the flag. Both are acceptable outcomes of this issue; what remains is whether plugins should move to the first for parity.

Additional context

Introduced with keyless-by-default push signing for skills (#6383, #6385, #6390). #6438 mirrored plugin push from the skills surface but deliberately diverged on this one point, omitting Key rather than reproducing the dead end.

The interim mitigation is now a permanent-until-decided contract: cmd/thv/app/ai_plugin_push.go and the plugins trust-model doc both state that plugin signing is keyless-only and why. The skills --key help text has since been updated to name the --public-key step, which the original version of this issue flagged as missing.

Discovered during review of #6438 (thanks @JAORMX).

Related: #6300 (plugin Sigstore stack, closed 2026-09-03), #6307 (keyless push signing), #5740 (RFC THV-0080).

[samuv]

Scope update — the plugins side of this changed in #6438.

Rather than ship a documented trap, --key was removed from thv ai-plugin push entirely (commit e2f2055d8): the flag, the key field on the push API DTO and Go client, and the forwarding to signer.Options are all gone, and pluginsvc.Push returns a 400 naming the reason if an in-process caller sets PushOptions.Key (the field still exists, since plugins.PushOptions aliases skills.PushOptions). Plugin publishing is now keyless or explicitly --no-sign.

So the work here is now asymmetric between the two surfaces:

• Skills: thv skill push --key still accepts a key and still produces an uninstallable artifact. This is the live bug, and it is currently undocumented — the interim warning that briefly existed was on the plugin flag, which no longer exists.
• Plugins: no bug to fix, but --key needs re-adding to thv ai-plugin push (plus the DTO field, the client field, and removing the guard in validateSigningInputs) as part of whatever lands key verification.

The core work is unchanged and still shared: a public key on the lock entry, install options, and the install API, wired to the already-unused VerifyOCIWithKey and VerifyBundleOfflineWithKey, with offline sync re-verification going through VerifyBundleOfflineWithKey.

Suggested sequencing, since skills is the one currently exposed: land the shared plumbing and fix skills first, then re-add the plugin flag on top — the plugin side is a strict superset of the skills change and has no user-visible gap in the meantime.

[samuv]

Design resolved — supporting key verification, install-only for v1

Worked through this in detail. Landing on the first arm ("support it"), scoped to install for v1. Recording the decisions and, more importantly, the five code facts that each invalidated an obvious-looking approach.

Decisions

Decision	Resolution
Arm	Support key verification. --key stays on push
Trust anchor	Full key material in the lock; --public-key mandatory on true first use, TOFU thereafter
Lock field	Provenance.publicKey, single-line base64 DER SPKI, mutually exclusive with signerIdentity/certIssuer
Lock schema version	No bump
Dispatch	Lock-first — the expectation picks the verify path, never the artifact
Detection	coreverifier.Bundle.HasCertificate()
Messaging	New verifier.ErrKeySigned; --allow-unsigned does not override it
Transport	CLI reads the PEM and sends the key material (not a path)
Conflicting supplied key	Rejected up front, never silently ignored
Re-anchoring in v1	No in-place path — uninstall, then reinstall with --public-key
toolhive-core	No change or release needed

Five facts that shaped it

1. Key signing writes no Rekor entry. toolhive-core/container/signer/signer.go:99 — "FulcioURL and RekorURL apply to keyless signing only and are ignored for the key-pair flow." A --key bundle is a bare detached signature: no certificate, no transparency log, therefore no signing time and no revocation. Any key entry is a strictly weaker trust tier than a keyless one, and cannot participate in checkPinnedCertificateFields, catalog-declared provenance (Catalog-supplied expected identity, so first install is not trust-on-first-use #6310), or identity-based signer-change detection.

2. The public key cannot be recovered from the artifact or the stored bundle. signatureAnnotations (cosign_attach.go:236) writes only signature, certificate, and rekor-bundle annotations — there is no annotation for the key — and the retrieve side reconstructs the bundle with a fixed placeholder, keySignedPublicKeyHint = "cosign-keypair" (container/verifier/sigstore.go:219). Consequences: storing only a sha256: digest in the lock is impossible (nothing to hash at verify time), resultFromKey must not grow an observed-key field, and "wrong key" vs. "broken signature" are genuinely indistinguishable — so there is no ErrKeyMismatch worth adding.

3. Do not bump lockfile.CurrentVersion. validation.go:35 compares lf.Version != CurrentVersion, not <=, so bumping to 2 would make every existing v1 lock file unreadable by new thv. It is also unnecessary: validateProvenance already hard-requires signerIdentity, so an older thv reading a key-pinned entry fails the whole lock closed. The downgrade guard is already in place, for free.

4. validateProvenance rejects whitespace and non-graphic runes, so raw multi-line PEM cannot be stored — hence base64 DER SPKI, re-wrapped to PEM at verify time.

5. Core already shipped the discriminator. Bundle.HasCertificate() (container/verifier/bundles.go:63) is exported and documented for exactly this, with zero callers in toolhive. Along with VerifyOCIWithKey and VerifyBundleOfflineWithKey, that is three unused primitives anticipating this feature — though they are only ~35 of ~1400 lines, and the two verify functions still need signature changes.

Two adjacent paths already broken, not previously noted here

Both are in scope for this work but neither was in the original description:

• Upgrade misreports. probeCandidateSigner (skillsvc/upgrade.go:275) calls VerifyOCI(..., nil) — keyless-only. A key-signed candidate yields ErrSignatureInvalid, which guardSignerChange routes to UpgradeStatusFailed rather than SignerChangeBlocked: an unexplained failure where a policy decision was intended.
• Adoption is impossible. adoptSkill/adoptLocked back-fill trust via ResultFromBundle → observedFromResult, which requires a certificate. A key-signed install therefore cannot be adopted at all.

Transition semantics (v1)

Transition	Behavior
key → same key	allowed, silent
key → different key	SignerChangeBlocked; no in-place re-anchor
key → keyless	SignerChangeBlocked; --allow-signer-change suffices (candidate is chain-verifiable)
keyless → key	SignerChangeBlocked; no in-place re-anchor

--allow-signer-change must never re-anchor a key by itself. Its contract is "verify chain-of-trust only and re-record the observed identity" — for keyless, Fulcio and Rekor are the trust root, so that is self-sufficient; for a key there is no trust root at all, making it a null operation. Re-anchoring a key is semantically a first-use key install and must demand the same input.

Deferred to a follow-up (option A)

--public-key on upgrade and sync --adopt, plus an in-place re-anchor path requiring both --allow-signer-change and --public-key. Deferring these cuts the CLI surface from 6 touchpoints to 2 while keeping the trust model complete: both refusals above are honest fail-closed states, and v1 only makes their messages truthful.

PR sequence (~7 PRs, ~1400 lines, single repo)

1. Honest classification — ErrKeySigned via HasCertificate(); 403 with an actionable message; a test pinning that --allow-unsigned does not rescue it. Plus the independent gap noted in this issue: the missing caveat on thv skill push --key and in the skills arch doc.
2. Lock schema — Provenance.publicKey; required-fields becomes identity XOR key; distinct Info rendering so a key-pinned entry never renders as an identity-pinned entry with a blank signer.
3. Skills install — InstallOptions.PublicKey, API + Go client DTOs, --public-key, verifyOCIInstall key branch, conflict rejection.
4. Skills sync/upgrade — verifyStoredSignature key branch; adopt refuses with an actionable message; upgrade returns SignerChangeBlocked instead of an opaque failure.
5. Plugins install — mirror of 3.
6. Plugins sync/upgrade — mirror of 4.
7. E2E both surfaces + arch docs.

PR 1 is independently valuable: it converts today's inexplicable 403 into an accurate diagnosis and fixes the skills-side documentation gap, without touching the lock schema or any DTO. It is written and in verification now.