Upstream TLS trust: no config surface for a private CA, and no verification override

[janiussyafiq]

Description

Outbound TLS trust has no configuration surface. There is no ssl_verify-style switch and no way to point the gateway at a private CA bundle from config:

• no tls / ca_file / verify field in the upstream: config section, which otherwise carries the connection-layer settings (timeouts, keepalive, pool, retries);

• no TLS field on any resource in schemas/resources/ — in particular provider_key, which is where an upstream endpoint (api_base) is declared;

• AISIX_UPSTREAM_SSL_VERIFY is rejected at boot:

  unknown field `upstream_ssl_verify`, expected one of `etcd`, `resources_file`,
  `proxy`, `admin`, `observability`, `cache`, `ratelimit`, `upstream`,
  `downstream`, `managed`, `bedrock_endpoint_url`
  

A private CA can be trusted today, but only as an OS-level side effect and only process-wide: SSL_CERT_FILE / SSL_CERT_DIR are honoured, and are additive to the built-in roots (setting SSL_CERT_FILE to a private CA alone still leaves public providers trusted). Neither is documented.

Requested, in order:

1. Document the mechanism that already works — SSL_CERT_FILE / SSL_CERT_DIR, and the fact that it is additive. Zero code.
2. A first-class config surface for a private CA, so trust is declared where the endpoint is declared rather than in the process environment — a deployment-level upstream.tls.ca_file, and/or a per-provider_key CA for deployments facing more than one private CA.
3. An explicit, scoped verification override for test environments, as the peers below provide. Not global, and not the default.

Why

Self-hosted model endpoints and internal OIDC providers behind an enterprise CA are routine in on-prem deployments, and every outbound path is affected — the provider bridges, guardrails, MCP OAuth, A2A, OTLP export, and JWKS/discovery all share aisix_gateway::client_builder() (crates/aisix-gateway/src/upstream_http.rs:121).

The current answer is an environment variable that does not appear in the docs, the example config, or the resource schemas, so it is not discoverable from the failure. What operators see is a generic 502:

transport error: error sending request for url (https://...): client error (Connect): invalid peer certificate: UnknownIssuer

Worth flagging for whoever picks this up: the intuitive fix — dropping the CA in /usr/local/share/ca-certificates and running update-ca-certificates — does work, but only as root. The image runs as user aisix, so the command fails with Permission denied, leaves the bundle unchanged, and the gateway still rejects the certificate. It looks like the CA was installed when it was not.

Priority

Medium. A working workaround exists, but it is undocumented, process-wide, and cannot express "this one upstream, this one CA".

Reproduction

ghcr.io/api7/aisix:0.6.0, standalone (resources_file), against an HTTPS upstream whose certificate is signed by a private CA:

provider_keys:
  - display_name: private-pk
    provider: openai
    api_key: sk-mock
    api_base: https://internal-llm.example:8443/v1

Setting	Result
(none)	502 invalid peer certificate: UnknownIssuer
SSL_CERT_FILE=/certs/ca.crt	200
SSL_CERT_DIR=/certs	200
CA into system store + update-ca-certificates, as root	200
CA into system store, as the default aisix user	502 — update-ca-certificates silently fails
REQUESTS_CA_BUNDLE=/certs/ca.crt	502

---

Prior art

Product	Has it	Reference
LiteLLM	Yes	ssl_verify: false or a CA path, plus SSL_VERIFY / SSL_CERT_FILE
Bifrost	Yes	ca_cert_pem and insecure_skip_verify in provider network_config
Kong Gateway	Yes	ca_certificates, tls_verify, tls_verify_depth on the Service entity
Envoy AI Gateway	Yes	BackendTLSPolicy.validation.caCertificateRefs
Apache APISIX	Partial	plugin-level ssl_verify, default true since 3.16.0; upstream server certs are not verified at all, and upstream tls.verify is Kafka-only
Cloudflare AI Gateway	N/A	managed edge service reached by base-URL redirect
Portkey	Unknown	not researched

Every self-hostable peer exposes at least a CA bundle; three of four also expose an explicit skip.

_{Surveyed 2026-07-31; every claim rests on a fetched docs/source page.}

[coderabbitai]

🔗 Related PRs

#808 - fix(upstream): name the real transport fault and bound the connection layer [merged]
#825 - feat(auth): inbound OIDC/JWT authentication — trust providers + jwt_subject key binding [merged]
#835 - feat(headers): request-context variables in default_headers + client-header forwarding allowlist [merged]
#843 - docs: bring the README in line with what the gateway actually ships [merged]
#859 - feat(mcp): OAuth 2.1 resource-server discovery surface for /mcp [open]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[jarvis9443]

Shipped across four PRs. Both data-plane halves are merged; the control-plane and docs PRs are open.

Where trust is configured now

upstream.tls (ca_file, client_cert_file, client_key_file, verify) applies to every connection the gateway opens on a request path — the provider bridges, all seven guardrails, MCP OpenAPI/OAuth, A2A, passthrough, JWKS/OIDC discovery, OTLP export, the Realtime WebSocket, Bedrock, and the S3/GCS/Azure log-export stores. ca_file is additive, so trusting a private CA never makes a public provider unreachable.

provider_key.tls (ca_cert inline PEM, verify) covers the "more than one private CA" case from the report, declared on the key that carries api_base. A certificate on one key does not leak trust to another — that's the load-bearing E2E assertion.

Redis got its own tls block rather than sharing upstream.tls: the cache/rate-limit backend sits inside the deployment and is normally issued by a different authority than the model endpoints.

Three defects surfaced while wiring this up

• SSL_CERT_FILE worked only by accident. reqwest's rustls-tls-native-roots was never declared — it arrived through object_store's aws feature. Dropping the log-export sink would have silently stopped the gateway trusting anything the OS trusts. Now declared outright.
• rediss:// could not connect at all. The redis dependency enabled no TLS feature, so the URL was rejected at parse time with can't connect with TLS, the feature is not enabled; the TlsMode handling in sentinel mode was unreachable code.
• rustls::ClientConfig::builder() panics when several crypto providers are compiled in, which enabling the redis TLS feature makes true. The shared config now selects the provider explicitly.

Two limits, stated rather than silent

Bedrock can't express verify: false or a client certificate — the AWS SDK's stable HTTP surface is trust roots only, and the API that would allow it is #[doc(hidden)]. ca_file does apply. Redis in Sentinel mode likewise takes no custom roots, because the client library offers no way to pass them for the discovered master. Both log a warning when the setting is present.

On the update-ca-certificates trap you flagged — documented in docs#1991, including that it fails silently as the unprivileged aisix user and looks as though it succeeded.