diff --git a/pip/pip-478.md b/pip/pip-478.md new file mode 100644 index 0000000000000..b31696c9c3d30 --- /dev/null +++ b/pip/pip-478.md @@ -0,0 +1,857 @@ +# PIP-478: Asynchronous v5 client authentication plugin interfaces and TLS material provider plugin interface + +# Background knowledge + +Apache Pulsar's Java client lets applications plug in custom authentication through `org.apache.pulsar.client.api.Authentication` and `org.apache.pulsar.client.api.AuthenticationDataProvider`. The same pair has been in place since the first 2.x releases and is the basis of every client-side auth mechanism Pulsar ships (token, mTLS, KeyStore TLS, Basic, OAuth2, Athenz, SASL). + +**PIP-97** (*Asynchronous Authentication Provider*, accepted for 2.10/2.12) made the **broker-side** authentication framework asynchronous so that providers can do non-blocking I/O during authentication — for example, the OAuth2 / OIDC verifier needs to call out to an identity provider. PIP-97 deliberately scoped itself to the broker side and explicitly deferred the client side: deprecated methods on `Authentication` were left behind for later cleanup. Three years on, that cleanup never happened, and the deprecated methods (`getAuthData()` no-arg, `configure(Map)`) are still part of the v4 surface. + +**PIP-337** (*SSL Factory Plugin*, Pulsar 3.x) introduced a pluggable `PulsarSslFactory` so operators can replace Pulsar's default file-based TLS material loading with custom logic (e.g., using a Key Management System (KMS) API). The factory takes a `PulsarSslConfiguration` value object that includes — among the TLS file paths — a direct field `AuthenticationDataProvider authData`. This forced the SSL layer in `pulsar-common` to depend on the v4 `AuthenticationDataProvider` interface, even though the SSL factory ought to be a transport concern independent of who supplies the credentials. The PIP-337 classes live in `org.apache.pulsar.common.util`, alongside dozens of unrelated utility helpers, which is the wrong package for what is now a public SPI. + +**PIP-466** (*New Java Client API V5*, accepted for Pulsar 5.0) created two new modules — `pulsar-client-api-v5` and `pulsar-client-v5` — to host a modernised Java client API. PIP-466 is explicitly *additive*: the existing v4 modules (`pulsar-client-api`, `pulsar-client`) remain unchanged. PIP-466 sketched a sync `Authentication` stub in `pulsar-client-api-v5` and a basic adapter in `pulsar-client-v5`; this PIP replaces both with the async, capability-segregated SPI and the bridges described below. PIP-466 punted on auth design and explicitly invited a follow-up proposal to fill it in. This PIP is that follow-up. + +## How Pulsar client authentication works + +Pulsar clients authenticate to the broker over **two transports**, each with their own protocol semantics. The same authentication plugin must support both: applications typically produce / consume on the binary protocol AND administer the cluster through the HTTP REST API with the same credentials. + +### Transport A — Pulsar binary protocol + +The Pulsar binary protocol is a length-prefixed Protocol Buffers framing used for produce, consume, lookup, and admin RPCs over TCP (typically port 6650; 6651 for TLS). Authentication is conveyed in dedicated `Command*` messages: + +1. Client opens a TCP (or TLS) connection. +2. Client sends `org.apache.pulsar.common.api.proto.CommandConnect` carrying: + - `auth_method_name` — stable string identifying the plugin (e.g., `"token"`, `"basic"`, `"sasl"`). + - `auth_data` — opaque byte payload (the credential bytes — a JWT token, a SASL initial frame, etc.). +3. Broker either: + - **Succeeds immediately** (single-pass authentication): broker sends `CommandConnected`. The connection is ready. + - **Sends a challenge** (multi-round authentication): broker sends `CommandAuthChallenge` carrying a `challenge` payload. Client replies with `CommandAuthResponse` carrying response bytes. This repeats until the broker accepts (then sends `CommandConnected`) or rejects. + - **Fails**: broker sends `ServerError` with `AuthenticationError`. The connection is closed. + +The binary protocol also reuses `CommandAuthChallenge` for **broker-pushed credential refresh**: a magic payload `AuthData.REFRESH_AUTH_DATA_BYTES = "PulsarAuthRefresh"` signals "your short-lived credential is about to expire (or has already expired) — produce a fresh one and respond with `CommandAuthResponse`." This is how OAuth2 / Athenz / other short-lived-credential schemes stay valid across long-lived connections. + +### Transport B — HTTP/HTTPS REST API (admin client, HTTP topic lookup) + +The Pulsar admin client and HTTP-mode topic lookup speak HTTP/1.1 (typically port 8080; 8443 for TLS). Authentication rides in standard HTTP headers: + +1. Client constructs an HTTP request. +2. Client attaches authentication headers. For single-pass credentials, typically `Authorization: ` — e.g., JWT auth sends `Authorization: Bearer `; Basic auth sends `Authorization: Basic `; Athenz sends a custom `Athenz-Role-Auth: ` header. +3. Server either: + - **Succeeds**: handles the request normally (2xx response). + - **Sends a challenge** (multi-round): replies with `401 Unauthorized` carrying the challenge in Pulsar-specific SASL headers — `SASL-Token` (the base64 token bytes), `State`, and `SASL-Server-ID` (to correlate the multi-round session) — **not** the standard `WWW-Authenticate` header. The client resubmits the request with updated `SASL-Token` / `State` / `SASL-Server-ID` headers. This repeats until the server accepts, at which point it responds `200 OK` carrying the final `SaslAuthRoleToken`. SASL (Kerberos) over HTTP follows this pattern, driven server-side by `AuthenticationProviderSasl` and client-side by `AuthenticationSasl`. + - **Fails**: the request is rejected (`401`/`403`) and not retried. + +Today this multi-round HTTP loop is **not** driven by a generic framework component — the SASL plugin implements it itself. `AuthenticationSasl.authenticationStage(...)` recursively resubmits the request on each `401` until the server returns `200`, and both the admin client and the HTTP-lookup client simply call the plugin's `authenticationStage(...)` / `newRequestHeader(...)` hooks. Two quirks follow from this: the SASL stage always re-issues the exchange as a `GET` to the original URI (even when the original request was a `POST`), and the admin client and HTTP-lookup client reach the same plugin through two different HTTP stacks — JAX-RS for the admin client, AsyncHttpClient for HTTP lookup. The v5 design relocates this loop into framework-side **drivers** — one per HTTP stack — leaving the plugin to compute only each round's response; see [the capability-segregated SPI](#capability-segregated-authentication-spi). + +### Authentication styles + +Independent of transport, every Pulsar authentication method follows one of two styles: + +- **Single-pass credential exchange.** The client presents a credential (a token, a username/password pair, a signed assertion) and the broker accepts or rejects in a single round trip. Long-lived credentials need no refresh at all — a static JWT (`Token`) or a `Basic` username/password is presented as-is on every connection. Short-lived credentials *do* expire, and refreshing them is an **internal concern of the authentication plugin**: the plugin produces a fresh credential when next asked (and may proactively renew in the background, or react to the broker's refresh sentinel — see Transport A). The framework does not manage refresh; it simply re-invokes the plugin's async credential method. Examples: `Token`/JWT and `Basic` (long-lived, no refresh); OAuth2 (short-lived access token the plugin renews against the IdP) and Athenz (short-lived role tokens the plugin renews against ZTS). + +- **Multi-round challenge/response.** The client and broker exchange multiple frames before the broker grants access. The exchanged bytes are opaque to the framework; only the implementation knows how to interpret a challenge and compose a response. Example: SASL (Kerberos). + +### A note on mTLS + +Mutual TLS (mTLS) is sometimes called an "authentication method", but the TLS material and handshake are a **transport-layer concern** rather than a credential-carrying authentication plugin (in v5 the mTLS case is handled by the built-in `TlsAuthentication` plugin — see below). The client's certificate and private key are configured at the `PulsarClient` builder level and attached to both transports' `SSLContext`s. The broker authenticates the client by inspecting the certificate chain presented during the TLS handshake (its server-side `AuthenticationProviderTls` reads `SSLContext.getSession()`). No `auth_data` payload is exchanged at the Pulsar protocol layer. + +The v4 client conflated this by exposing `AuthenticationTls` as an `Authentication` plugin whose main purpose was to point the SSL layer at certificate files. **In v5, the TLS key and certificate for mTLS are configured directly at the `PulsarClient` builder**, not in an authentication plugin. mTLS authentication is then represented by the built-in `TlsAuthentication` plugin, which carries no TLS material — it only makes the binary protocol set `auth_method_name=tls` on `CommandConnect` (the broker reads the client certificate from the TLS handshake). The compatibility layer maps legacy v4 `AuthenticationTls` configuration onto this builder-level TLS material **plus** the built-in `TlsAuthentication` plugin, and the same implementation backs both the v4 and v5 client APIs in Pulsar 5.0. + +### Concrete examples + +| Method | Style | Binary `auth_data` | HTTP encoding | Notes | +|---|---|---|---|-------------------------------------------------------------------------------------| +| Token / JWT | single-pass | `utf8(jwt)` | `Authorization: Bearer ` | Most common deployment | +| Basic | single-pass | `utf8(user:pass)` | `Authorization: Basic ` | | +| OAuth2 | single-pass | `utf8(access_token)` | `Authorization: Bearer ` | Token-exchange step uses HTTP to the IdP | +| Athenz | single-pass | `utf8(role_token)` | custom `Athenz-Role-Auth: ` header | ZTS exchange uses Athenz SDK | +| SASL | multi-round | `sasl_initial_bytes`, then `CommandAuthResponse` rounds | custom `SASL-Token` / `State` / `SASL-Server-ID` headers, multi-round via repeated `401` (no `WWW-Authenticate`) | | +| mTLS | transport-layer | empty (`auth_method_name=tls`) | n/a (cert presented at TLS handshake) | TLS material configured at the builder; the built-in `TlsAuthentication` plugin sets `auth_method_name=tls` on the binary protocol | + +### Key concepts used in this proposal + +- *Capability segregation* — an interface describes one cross-cutting concern; an implementation that needs to expose several concerns implements several interfaces. The opposite is the *kitchen-sink* shape, where a single interface declares every concern and implementations leave most methods returning `null`. + +- *Async authentication* — completing authentication-related work via `CompletableFuture` so the calling thread (typically a Netty I/O thread on the connection path) is not blocked while the auth provider performs other I/O (e.g., calling a remote token endpoint). + +- *Single-pass vs challenge-response* — the two authentication styles described above. Reflected in the v5 interfaces. + +- *Configuration vs initialization* — in v5 these are separate concerns. **Configuration** is supplying the plugin its `authParams` (the path to the token file, the OAuth2 issuer URL, etc.) — static, plugin-specific data, independent of when/where the plugin runs. **Initialization** is when the plugin is given runtime services (a `PulsarHttpClient` factory, a scheduler, an OpenTelemetry handle) and may do I/O to prepare itself. The v4 interface conflated these with `configure(Map)` + `start()`; v5 splits them cleanly so the same plugin instance can be constructed and configured at one time and initialized later when a `PulsarClient` is built. + +# Motivation + +The v4 client authentication surface has five concrete, observable problems that are not addressed by any in-flight work. + +### 1. The Netty event loop blocks under token refresh. + +`ClientCnx.handleAuthChallenge()` is invoked when the broker pushes `CommandAuthChallenge`. The broker also uses this command, with payload `AuthData.REFRESH_AUTH_DATA_BYTES`, to signal "your short-lived credential is about to expire or has expired — fetch a new one and reply." `ClientCnx` services this push synchronously: it calls `authentication.getAuthData(remoteHostName)` and then `authenticationDataProvider.authenticate(challenge)` on whatever thread delivered the channel-read event — namely, the Netty I/O loop. For `AuthenticationToken` and `AuthenticationTls` this is fast, but for `AuthenticationOAuth2` it can stall the loop while a token endpoint is hit, and for `AuthenticationAthenz` while ZTS is queried. A single slow refresh therefore halts every I/O multiplexed onto that loop — producers, consumers, lookups — across every connection. + +### 2. `AuthenticationDataProvider` is a kitchen sink. + +A single interface declares 13 methods covering three unrelated concerns: TLS material (cert chain, private key, file paths, keystore params, truststore stream — 7 methods), HTTP request authentication (auth-type, headers — 3 methods), and binary-protocol authentication (command data, plus SASL `authenticate(AuthData)` for challenge-response — 3 methods). Every built-in implementation returns `null` or `false` from the methods it does not care about. Implementers cannot tell at compile time which transports an `Authentication` actually supports — the answer is "whichever combination of getters happen to return non-null at runtime." There is no compile-time guarantee that an impl claiming `hasDataForTls()` will return non-null from `getTlsCertificates()`, and no clean way to write a composite implementation (say, mTLS *and* OAuth) without re-hashing this implicit contract. + +### 3. Authentication plugins such as the plugin for OAuth2 cannot share resources with the rest of the runtime. + +`AuthenticationOAuth2` needs HTTP to talk to its identity provider. Today the OAuth2 implementation creates *three separate* `DefaultAsyncHttpClient` instances (in `FlowBase`, `TokenClient`, and `DefaultMetadataResolver`) because the API surface gives it nowhere to ask "the client" for an HTTP client. `FlowBase` even constructs a private `PulsarSslFactory` instance to configure those clients. Apache Pulsar issue [#24795](https://github.com/apache/pulsar/issues/24795) is exactly this complaint — operators want to control the HTTP client (proxies, retry policy, observability) without forking — and PR [#24944](https://github.com/apache/pulsar/pull/24944) is a workaround patch for one specific case. The root cause is that the `Authentication` SPI gives implementations no hook to acquire shared client services. Beyond a plain HTTP client, plugins also need to share the Netty DNS-resolution configuration and DNS cache. + +### 4. The PIP-337 SSL Factory interface is a kitchen sink that exposes implementation details and prevents a clean separation of concerns. + +`PulsarSslConfiguration` — the value object that `PulsarSslFactory` implementations consume — carries a field `AuthenticationDataProvider authData`, which forces the SSL layer in `pulsar-common` to depend on the entire v4 auth SPI even though it only needs TLS material. This coupling exists because v4 `AuthenticationTls` could override the client's TLS configuration, and that override was wired through `PulsarSslConfiguration` instead of a clean integration point. + +The interface itself is also a kitchen sink: `initialize(PulsarSslConfiguration)`, `needsUpdate()`, `update()`, `createInternalSslContext()`, `getInternalSslContext()`, and `getInternalNettySslContext()` expose implementation details — the order in which a default file-based implementation rebuilds its state on rotation — as public methods. A custom plugin should not have to deal with any of this; it should only supply the TLS material and configuration for the use case it supports (for example "TLS for the Pulsar binary protocol client", "TLS for the admin HTTP client", or "TLS for OAuth2 token-endpoint calls"), leaving the framework to build, cache, and rotate the `SSLEngine`/`SSLContext`. Finally, the SPI is synchronous, so loading TLS material can block the Netty I/O loop — the same hazard as Motivation #1. + +### 5. Challenge-response authentication is a maintenance burden because its handling is scattered across the stack. + +Multi-round (challenge/response) authentication has no single integration point, so its logic is duplicated and entangled with unrelated concerns. On the binary protocol, `ClientCnx.handleAuthChallenge()` interleaves the initial connect, broker-pushed credential refresh, and SASL challenge rounds on one synchronous path. On HTTP there is no generic multi-round driver at all: the SASL plugin implements the `401`→resubmit→`200` loop itself (`AuthenticationSasl.authenticationStage(...)`), and the admin (JAX-RS) client and the HTTP-lookup client each drive it separately, so the loop is effectively re-implemented per HTTP stack. The SASL plugin even takes over request construction, re-issuing the exchange as a `GET` to the original URI regardless of the original method. The result is costly to maintain and hard to extend: supporting another challenge-response scheme (such as HTTP digest) would mean changing the plugin, both HTTP client stacks, and the binary connect path instead of implementing one well-defined capability. + +# Goals + +## In Scope + +This PIP introduces, in the existing `pulsar-client-api-v5` and `pulsar-client-v5` modules created by PIP-466: + +1. **Asynchronous, capability-segregated v5 authentication SPI** in `org.apache.pulsar.client.api.v5.auth` +2. **Two configuration paths** matching the existing v4 idioms: + - **Programmatic**: user constructs an `Authentication` instance (typically via a constructor or builder) and passes it to `PulsarClient.builder().authentication(myAuth)`. `configure(...)` is NOT called by the framework in this path (the instance is assumed already configured by the caller). + - **String-based**: user supplies `authPluginClassName` + `authParams` (JSON map or existing `key:val,key:val` String format). The framework reflectively instantiates the class via its no-arg constructor, calls `configure(parsedAuthParams)` once, then `initializeAsync(ctx)` once. Matches the existing `AuthenticationUtil.create(...)` semantics. + +3. **`PulsarHttpClient` SPI** in `org.apache.pulsar.client.api.v5.http`, with **framework-managed lifecycle**. The framework owns Netty event loops, timers, DNS caches, and TLS material refresh integration; plugins describe what kind of HTTP client they need via a `PulsarHttpClientConfig` (timeouts, proxy, optional dedicated TLS credentials) and obtain an instance via `AuthenticationInitContext.httpClientFactory().newHttpClient(config)`. The framework MAY issue multiple `PulsarHttpClient` instances per `PulsarClient` for different uses — OAuth2's mTLS exchange to the IdP, HTTP topic lookup, and an admin client may all need different TLS configurations, but they share the underlying event loop / timer / DNS resources. Default implementation backed by AsyncHttpClient, discovered via `ServiceLoader`. + +4. **Redesigned PIP-337 SSL provider** — purpose-driven, no kitchen-sink config. The existing `PulsarSslFactory` / `PulsarSslConfiguration` surface is completely removed and replaced in Pulsar 5.0 and in v5 client's TLS configuration and integration points. This change also impacts the broker side. + +5. **v5 client builder additions for mTLS** at the client configuration level (mTLS may also be used for TLS auth). + +6. **Compatibility bridge** in `pulsar-client-v5`, in both directions: + - `LegacyV4AuthenticationAdapter` — wraps a v4 `Authentication` instance as a v5 `Authentication` declaring the right capability for its style. v4 calls are always off-loaded to `ctx.scheduler()`; the Netty event loop never runs the v4 plugin's I/O. + - `V5ToV4AuthenticationAdapter` — exposes a v5 `Authentication` through the v4 `Authentication` interface that `ClientCnx` already drives, so the v5 SPI can back the v4 client API and built-in v4 shims. + +7. **Pulsar 5.0 client's internal implementation migration to use the v5 Authentication SPI and TLS SPI**. This applies also to the v4 client API usage in Pulsar 5.0 client since it provides both v4 and v5 client APIs. + +## Out of Scope + +- **Broker-side authentication.** Pulsar's broker-side `AuthenticationProvider` / `AuthenticationState` interfaces have their own set of design problems (PIP-97 only covered the async basics). They are out of scope; a sibling PIP will address them with the same design principles. + +- **Removal of the v4 `Authentication` interface or its deprecated methods.** The v4 surface — including the deprecated `getAuthData()` no-arg and `configure(Map)` methods — is retained indefinitely for source compatibility. Their bodies are re-implemented on top of the v5 SPI as part of in-scope item #7, but the method signatures stay. + +- **Non-Java client SDKs.** Each non-Java SDK (Python, Go, C++, Node.js) follows its own auth model and will be addressed by per-SDK PIPs. + +# High Level Design + +This proposal centers on a small core `Authentication` interface plus four narrow, opt-in **capability interfaces**, a framework-managed `PulsarHttpClient` SPI, and a purpose-driven `PulsarTlsMaterialProvider` SPI that replaces PIP-337. This section describes the shape and the reasoning; the full type listings live in the [Detailed Design](#detailed-design). + +## Capability-segregated authentication SPI + +Every plugin implements the core `Authentication` interface, which carries only lifecycle: a `configure(Map)` hook (called for the string-based path only), an `initializeAsync(AuthenticationInitContext)` hook that may do I/O, a `capability(Class)` lookup for delegating wrappers, and `close()`. + +The actual credential work lives on **four capability interfaces**, segregated by transport (Pulsar binary protocol vs HTTP) and by authentication style (single-pass vs multi-round challenge/response). A plugin implements only the ones it supports: + +- `BinaryProtocolAuthDataProvider` — single-pass credential for the binary protocol. +- `HttpAuthHeadersProvider` — single-pass credential for HTTP. +- `ChallengeResponseHandler` — multi-round challenge/response for the binary protocol. +- `HttpHeaderChallengeResponseHandler` — multi-round challenge/response for HTTP in the **SASL style** (repeated `401` carrying Pulsar's custom SASL headers, as used by the existing SASL-over-HTTP mechanism). + +These four are the integration points today's protocols need; the model is open-ended. Additional capability interfaces can be introduced later if a future mechanism needs to plug into the binary or HTTP authentication flow in a way these four don't capture, without disturbing existing plugins. HTTP multi-round auth is the first place this is expected: the SASL style above is custom to Pulsar, whereas standard `WWW-Authenticate`-based schemes (e.g. HTTP digest) follow a different exchange. The design therefore anticipates a **second, standard-style** HTTP challenge-response interface alongside `HttpHeaderChallengeResponseHandler`, with the framework's HTTP auth **driver** (below) selecting the handling by which interface a plugin implements. Only the SASL-style interface ships initially; the exact split is an [open question](#open-questions). + +**HTTP auth drivers.** Unlike the binary protocol — where `ClientCnx` is the single place that runs the `CommandAuthChallenge`/`CommandAuthResponse` loop — HTTP has two distinct client stacks (JAX-RS for the admin client, AsyncHttpClient for HTTP topic lookup; see [Transport B](#transport-b--httphttps-rest-api-admin-client-http-topic-lookup)). In v4 the multi-round loop lives inside the SASL plugin itself, which also re-issues the exchange as a `GET` to the original URI. v5 instead puts the loop in a framework-side **driver** — one implementation per HTTP stack — that runs the `401`→resubmit→`200` sequence and calls the plugin only to compute each round's response. For backward compatibility the SASL driver preserves the existing behaviour (custom headers, the `GET`-to-original-URI takeover); a future standard-style driver will handle `WWW-Authenticate`/digest. The driver is internal to the framework and is not part of the plugin SPI. + +For convenience, two **composite interfaces** bundle the common combinations so an implementation can declare one interface instead of several: + +- `SinglePassAuthentication extends Authentication, BinaryProtocolAuthDataProvider, HttpAuthHeadersProvider` — a single-pass credential served over both transports. +- `ChallengeResponseAuthentication extends Authentication, BinaryProtocolAuthDataProvider, HttpAuthHeadersProvider, ChallengeResponseHandler, HttpHeaderChallengeResponseHandler` — a plugin that supports both styles over both transports. + +Composites are purely a convenience; an implementation is free to implement the individual capability interfaces directly and combine only the ones it needs. + +**Built-in `TlsAuthentication` plugin (mTLS).** mTLS is not a capability that other plugins mix in — it is a small, self-contained `Authentication` implementation. The built-in `TlsAuthentication` class implements `Authentication` and `BinaryProtocolAuthDataProvider`, reporting `authMethodName() == "tls"` (overridable via its constructor) with empty `auth_data` so the binary protocol authenticates via the TLS handshake. It carries **no** TLS material — certificates and keys are configured at the `PulsarClient` builder (see [A note on mTLS](#a-note-on-mtls)) — and exists only so a client that wants mTLS auth can select it and have the binary protocol send `auth_method_name=tls` on `CommandConnect`. + +These interfaces replace v4's kitchen-sink `AuthenticationDataProvider`: capabilities make a plugin's supported transports and styles visible at compile time, and composing concerns (e.g., binary + HTTP single-pass) is a matter of implementing two interfaces rather than re-hashing an implicit contract. Capability discovery uses Java's natural mechanism — `instanceof` for direct implementations and the `capability(Class)` method for delegating wrappers; there is no registry. All capability methods are asynchronous (`CompletableFuture`), so credential acquisition never blocks the Netty event loop. + +Cross-round runtime state (a SASL conversation, a multi-stage HTTP handshake) lives on the per-call context's **state slot**, not on method parameters (see [per-call contexts](#authenticationcallcontext--httpauthcallcontext)). + +## Two configuration paths + +Both v4 idioms are preserved (see In-Scope item #2): the **programmatic** path, where the user constructs and configures the `Authentication` instance and `configure(...)` is not called by the framework; and the **string-based** path, where `authPluginClassName` + `authParams` drive reflective construction followed by one `configure(...)` and one `initializeAsync(...)`. + +## Initialization and per-call contexts + +`AuthenticationInitContext` is passed once to `initializeAsync(...)` and exposes the framework's shared runtime services: a `PulsarHttpClientFactory`, a `ScheduledExecutorService`, a `Clock`, an `OpenTelemetry` handle, the client instance id, and a convenience copy of the configured `params`. The framework owns and closes these shared services; the plugin may retain references for its lifetime and releases its own resources in `close()`. + +`AuthenticationCallContext` (binary) and `HttpAuthCallContext` (HTTP) are cheap per-call objects that carry transport-specific routing (broker host/port, or request URI) and expose a per-exchange **state slot** (`getStateObject`/`setStateObject`, keyed by class) so an implementation can retain conversation state across challenge-response rounds. The slot's lifetime equals one authentication exchange; concurrent authentications to different brokers each get their own context, so in-flight handshakes don't collide. (This state slot replaces an earlier-draft `previousResponseHeaders` parameter.) + +## The `PulsarHttpClient` SPI + +Auth plugins that need HTTP (OAuth2's token endpoint, Athenz's ZTS) obtain a client from the framework rather than constructing their own — fixing the v4 problem where `AuthenticationOAuth2` spins up three private `DefaultAsyncHttpClient` instances (Motivation #3). The **framework manages HTTP client lifecycle** (Netty event loop, timer, DNS cache, TLS material refresh); a plugin describes what it needs via a `PulsarHttpClientConfig` and receives an instance from `AuthenticationInitContext.httpClientFactory()`. + +The framework may hand out **multiple** `PulsarHttpClient` instances per `PulsarClient`: OAuth2's mTLS exchange to the IdP, HTTP topic lookup, and the admin client can each need a different TLS configuration (different trust domains), but they share the underlying event-loop / timer / DNS resources. The default backend is AsyncHttpClient, discovered via `ServiceLoader`; a provider is selected once per client (by name or by priority). + +## The `PulsarTlsMaterialProvider` SPI (PIP-337 replacement) + +PIP-337's `PulsarSslFactory` / `PulsarSslConfiguration` are removed entirely (Motivation #4) and replaced by a purpose-driven, material-focused SPI. A **`TlsPurposeContext`** identifies *why* TLS is requested and in what role — a `ClientTlsPurposeContext` (binary client, HTTP lookup, OAuth2/generic) or a `ServerTlsPurposeContext` (broker, proxy, web service, plus the advertised-listener name on the server side). The provider answers `getTlsMaterial(TlsPurposeContext)` with a `TlsMaterial` carrying the cert chain, private key, trust certs, protocols, ciphers, plus client/server-specific flags. The framework — not the plugin — turns `TlsMaterial` into an `SSLEngine`/`SSLContext` and caches it, rebuilding only when the material changes. The SPI deliberately omits PIP-337's implementation-detail methods (`needsUpdate()`, `update()`, `getInternalSslContext()`, …). + +The provider also drives a **`TlsMaterialListener`**: it notifies the listener once when the material for a purpose is first ready, then again whenever it changes. On the broker side the server must rebuild its listener `SSLContext` on rotation; on the client side the framework could either use the listener when suitable or re-request `getTlsMaterial(...)` when a new TLS connection is created. The default `FileBasedTlsMaterialProvider` loads PEM and keystore files and reloads on rotation, and supports **dynamic registration** of additional file-based material sources per purpose after construction (used by the v4 `AuthenticationTls` bridge); a custom provider can integrate a Key Management System (KMS) and may differ between client and broker sides. + +## Compatibility bridge for v4 plugins + +`LegacyV4AuthenticationAdapter` wraps an arbitrary v4 `Authentication` as a v5 `Authentication`, declaring the capability interfaces that match the v4 plugin's style. All v4 calls are off-loaded to `ctx.scheduler()` so the Netty event loop never runs v4 plugin I/O. v4 plugins that supply TLS material (`hasDataForTls() == true`, e.g., `AuthenticationTls`) have that material registered with the client's `PulsarTlsMaterialProvider` and are represented by the built-in `TlsAuthentication` plugin. + +## `ClientCnx` async-driver carve-out + +The only change to the otherwise-untouched v4 client is a marker interface `AsyncAuthenticationDriver` (in `pulsar-client-api`, `.internal` subpackage). `ClientCnx` detects it and routes connect / refresh / challenge handling through the async API; plain v4 instances keep the existing synchronous path verbatim. This is generic across all challenge types. + +## Error model + +All async failures complete the returned future exceptionally with a v5 `org.apache.pulsar.client.api.v5.PulsarClientException` — `AuthenticationException` (terminal), `GettingAuthenticationDataException` (transient/retryable), or `UnsupportedAuthenticationException` (capability not supported by a wrapped v4 impl). Details and the v4-exception translation are in the Detailed Design. + +# Detailed Design + +## Design & Implementation Details + +### The `Authentication` core and capability interfaces + +The new core `Authentication` interface lives in `pulsar-client-api-v5`: + +```java +package org.apache.pulsar.client.api.v5.auth; + +import java.util.Map; +import java.util.Optional; +import java.util.concurrent.CompletableFuture; + +public interface Authentication extends AutoCloseable { + /** + * Configuration step. Called once by the framework AFTER no-arg construction + * when the plugin was loaded reflectively from authPluginClassName + + * authParams. NOT called when the plugin was constructed programmatically + * by the user — that path presumes the instance is already configured. + * Default: no-op. Implementations override to read their parameters. + * + *

Configuration is intentionally separate from {@link #initializeAsync}: + * configuration is static plugin-level data (file paths, URLs, scopes); + * initialization gives the plugin runtime services and may do I/O. + */ + default void configure(Map authParams) {} + + /** + * Initialization step. Called once by the framework with runtime services + * after configuration. May do I/O; the returned future completes when the + * implementation is ready to serve credentials. + */ + CompletableFuture initializeAsync(AuthenticationInitContext ctx); + + /** Capability lookup for delegating wrappers. */ + default Optional capability(Class kind) { + return kind.isInstance(this) ? Optional.of(kind.cast(this)) : Optional.empty(); + } + + @Override default void close() throws Exception {} +} +``` + +The four capability interfaces, each in its own file in the same package — segregated by transport (binary vs HTTP) and style (single-pass vs challenge-response). An implementation declares the ones that match what it supports (e.g., a one-pass plugin that serves both transports implements both `BinaryProtocolAuthDataProvider` and `HttpAuthHeadersProvider`): + +```java +package org.apache.pulsar.client.api.v5.auth; + +import java.util.concurrent.CompletableFuture; + +/** Single-pass credential exchange for Pulsar binary protocol. */ +public interface BinaryProtocolAuthDataProvider { + /** Stable identifier sent in CommandConnect.auth_method_name */ + String authMethodName(); + + /** + * Produce a credential for the binary protocol connection. The returned + * {@link BinaryProtocolAuthData} carries the {@code auth_method_name} and the + * {@code auth_data} bytes for {@code CommandConnect}. + */ + CompletableFuture getAuthDataAsync(AuthenticationCallContext ctx); +} + +/** Single-pass credential exchange for Pulsar HTTP transport. */ +public interface HttpAuthHeadersProvider { + /** + * Produce the authentication headers for an outgoing HTTP request (e.g. + * {@code Authorization: Bearer }, or a custom header such as + * {@code Athenz-Role-Auth}). The returned {@link HttpAuthHeaders} are attached + * to the request; most implementations produce the same credential for every + * call. Completes exceptionally on failure (see the error model). + */ + CompletableFuture getHttpHeadersAsync(HttpAuthCallContext ctx); +} + +/** Multi-round challenge/response (SASL-style and custom protocols) for Pulsar binary protocol. */ +public interface ChallengeResponseHandler { + /** + * Respond to a binary-protocol {@code CommandAuthChallenge}. The framework places + * the returned bytes in {@code CommandAuthResponse}. Completion is decided by the + * broker (it replies with {@code CommandConnected} when satisfied), so the handler + * does not signal it; cross-round conversation state is kept in the context's + * state slot. + */ + CompletableFuture respondToChallengeAsync(AuthenticationCallContext ctx, + AuthChallenge authChallenge); +} + +/** + * SASL-style multi-round challenge/response over HTTP. Handles the existing Pulsar + * SASL-over-HTTP mechanism, where the server returns HTTP {@code 401} carrying the + * challenge in Pulsar's custom SASL headers (e.g. {@code SASL-Token} / {@code State} / + * {@code SASL-Server-ID}) rather than the standard {@code WWW-Authenticate} header. + * Standard {@code WWW-Authenticate}-based schemes (e.g. HTTP digest) will be served by a + * separate interface; the framework's HTTP auth driver dispatches to whichever a plugin + * implements. + */ +public interface HttpHeaderChallengeResponseHandler { + /** + * Compute the headers for the next round. The server's challenge headers from the + * prior {@code 401} are available via {@link HttpAuthCallContext#serverChallengeHeaders()}; + * the returned {@link HttpAuthHeaders} are attached to the resubmitted request. + * Cross-round conversation state is kept in the context's state slot. + */ + CompletableFuture respondToHttpChallengeAsync(HttpAuthCallContext ctx); +} +``` + +Two optional **composite interfaces** bundle the common combinations. They add no methods — an implementation may declare a composite instead of listing the individual capabilities, or implement the capability interfaces directly: + +```java +package org.apache.pulsar.client.api.v5.auth; + +/** Convenience: a single-pass credential served over both transports. */ +public interface SinglePassAuthentication + extends Authentication, BinaryProtocolAuthDataProvider, HttpAuthHeadersProvider {} + +/** Convenience: both authentication styles over both transports. */ +public interface ChallengeResponseAuthentication + extends Authentication, BinaryProtocolAuthDataProvider, HttpAuthHeadersProvider, + ChallengeResponseHandler, HttpHeaderChallengeResponseHandler {} +``` + +The built-in **`TlsAuthentication`** plugin handles mTLS. It is an ordinary `Authentication` implementation, **not** a capability interface: it implements `BinaryProtocolAuthDataProvider` with `authMethodName()` defaulting to `"tls"` and empty `auth_data`, carries no TLS material (that is configured at the builder), and exists only to drive `auth_method_name=tls` on the binary protocol so the broker authenticates from the TLS-handshake certificate: + +```java +package org.apache.pulsar.client.impl.v5.auth; + +import java.util.concurrent.CompletableFuture; +import org.apache.pulsar.client.api.v5.auth.*; + +/** + * Built-in mTLS plugin. The certificate and private key are configured at the + * {@code PulsarClient} builder (see "A note on mTLS"), not here. This plugin only + * reports {@code authMethodName()} (default {@code "tls"}) with empty {@code auth_data}, + * so the binary protocol sends {@code auth_method_name=tls} on {@code CommandConnect} + * and the broker authenticates from the certificate presented during the TLS handshake. + */ +public class TlsAuthentication implements Authentication, BinaryProtocolAuthDataProvider { + private final String authMethodName; + + public TlsAuthentication() { this("tls"); } + public TlsAuthentication(String authMethodName) { this.authMethodName = authMethodName; } + + @Override public String authMethodName() { return authMethodName; } + + @Override public CompletableFuture initializeAsync(AuthenticationInitContext ctx) { + return CompletableFuture.completedFuture(null); + } + + @Override public CompletableFuture getAuthDataAsync(AuthenticationCallContext ctx) { + return CompletableFuture.completedFuture(new DefaultBinaryProtocolAuthData(authMethodName, new byte[0])); + } +} +``` + +### Value types + +The credential/challenge value types in `org.apache.pulsar.client.api.v5.auth`. `BinaryProtocolAuthData` carries the binary-protocol credential; `HttpAuthHeaders` carries the HTTP headers; `AuthChallenge` / `ChallengeResponse` carry the multi-round binary exchange. + +```java +/** Binary-protocol credential: the auth-method name and the auth_data bytes. */ +public interface BinaryProtocolAuthData { + /** Auth-method name sent in CommandConnect.auth_method_name. */ + String authMethodName(); + /** Binary credential bytes sent in CommandConnect.auth_data. */ + byte[] authData(); +} + +/** Immutable default implementation of {@link BinaryProtocolAuthData}. */ +public record DefaultBinaryProtocolAuthData(String authMethodName, byte[] authData) + implements BinaryProtocolAuthData {} +``` + +```java +/** + * HTTP authentication headers an implementation produces for an outgoing request + * (and, for challenge-response, the headers carrying a server's challenge). Header + * names are canonicalised on construction (RFC 7230 §3.2); {@link #get} is + * case-insensitive; {@link #asMap} returns the canonical-cased view. + */ +public final class HttpAuthHeaders { + static HttpAuthHeaders empty(); + static HttpAuthHeaders of(String name, String value); + static HttpAuthHeaders of(Map headers); + Optional get(String name); // case-insensitive + Map asMap(); + // ... +} +``` + +```java +/** A binary-protocol CommandAuthChallenge payload handed to a ChallengeResponseHandler. */ +public interface AuthChallenge { + byte[] challenge(); +} + +/** + * Reply from respondToChallengeAsync — just the bytes for the next + * CommandAuthResponse. There is no completion flag: the broker decides when the + * handshake is finished (it sends CommandConnected); the implementation tracks any + * cross-round state of its own in the call context's state slot. Kept as a record + * rather than a bare byte[] so fields can be added later without an API break. + */ +public record ChallengeResponse(byte[] responseBytes) {} +``` + +### `AuthenticationInitContext` + +```java +public interface AuthenticationInitContext { + /** + * Factory for {@link PulsarHttpClient} instances. The framework manages + * lifecycle (event loop, timer, DNS cache, TLS material refresh integration); + * plugins describe what they need via {@link PulsarHttpClientConfig} and + * receive a configured instance owned by the framework. Multiple instances + * with different TLS / timeouts / proxy may be obtained for different uses + * (e.g., OAuth2 mTLS to the IdP vs HTTP topic lookup). + * + *

Plugins MUST NOT construct private HTTP clients directly — doing so + * defeats the framework's shared event-loop / DNS / refresh integration. + */ + PulsarHttpClientFactory httpClientFactory(); + + /** Executor for off-loop authentication work. Never the Netty event loop. */ + ScheduledExecutorService scheduler(); + + /** Used by implementations that schedule against wall-clock. */ + Clock clock(); + + /** Telemetry root; framework defaults to OpenTelemetry.noop() if unset. */ + OpenTelemetry openTelemetry(); + + /** Stable id of the owning PulsarClient for logging correlation. */ + String clientInstanceId(); + + /** + * Convenience copy of the parsed authParams previously passed to + * {@link Authentication#configure}. Mostly useful for plugins constructed + * reflectively that want their params at init time too. + */ + Map params(); +} +``` + +The lifecycle is: the framework constructs a single `AuthenticationInitContext` per `PulsarClient`, calls `initializeAsync(ctx)` once when the client is built, and the implementation may retain references for the lifetime of the client. `Authentication.close()` releases any resources the implementation acquired; the framework owns and closes the shared services (HTTP clients, scheduler). Per-call `AuthenticationCallContext` instances are cheap to allocate. + +### `AuthenticationCallContext` / `HttpAuthCallContext` + +A per-call context carries any transport-specific routing details, and exposes a **per-exchange state slot** for the implementation to retain conversation state across calls (challenge-response rounds, multi-stage HTTP auth, etc.): + +```java +public interface StatefulCallContext { + /** + * Retrieve an implementation-controlled state object previously stored + * with {@link #setStateObject}. The slot is keyed by class so multiple + * implementations can coexist without collision; impls typically store + * one object of their own type (e.g., their SASL conversation state). + * + *

The slot's lifetime equals the authentication exchange: for binary + * protocol, the lifetime of one {@code ClientCnx} setup (including all + * {@code CommandAuthChallenge}/{@code CommandAuthResponse} rounds); + * for HTTP, one request's retry sequence. Concurrent authentications + * to different brokers get their own context with their own slots, + * so multiple in-flight handshakes don't collide. + */ + Optional getStateObject(Class clazz); + + /** Store a state object keyed by its (or any) class. */ + void setStateObject(Class clazz, T value); +} +``` + +```java +public interface AuthenticationCallContext extends StatefulCallContext { + String brokerHost(); + int brokerPort(); +} +``` + +```java +public interface HttpAuthCallContext extends StatefulCallContext { + URI requestUri(); + + /** + * For SASL-style HTTP challenge/response: the challenge headers from the server's + * prior {@code 401} response (e.g. {@code SASL-Token} / {@code State} / + * {@code SASL-Server-ID}). Empty on the first request, before any challenge. + */ + Optional serverChallengeHeaders(); +} +``` + +Most implementations don't need to inspect any of the routing fields — they produce the same credential for every call. Implementations of `ChallengeResponseHandler` typically use the state slot to track their conversation across rounds (the server's challenge bytes arrive as the `AuthChallenge` parameter; cross-call state lives in `getStateObject(...)`). For HTTP, `HttpHeaderChallengeResponseHandler` reads the server's challenge from `HttpAuthCallContext.serverChallengeHeaders()` and keeps its own conversation state in the same state slot. + +### The `PulsarHttpClient` SPI + +The pluggable HTTP client lives in `org.apache.pulsar.client.api.v5.http`. The **framework manages HTTP client lifecycle**, including the Netty event loop, timer, DNS cache, and `PulsarTlsMaterialProvider` integration. Plugins describe what kind of HTTP client they need; the framework constructs, pools, and closes instances: + +```java +package org.apache.pulsar.client.api.v5.http; + +public interface PulsarHttpClient extends AutoCloseable { + CompletableFuture execute(HttpRequest request); + @Override default void close() {} +} + +/** + * Framework-owned factory. Plugins obtain HTTP clients via + * {@link AuthenticationInitContext#httpClientFactory()}; they MUST NOT + * construct private clients directly. Multiple instances per PulsarClient + * are supported — e.g., OAuth2 mTLS to the IdP uses a different + * TlsPurposeContext-driven TLS configuration than HTTP topic lookup, but they + * share the underlying event loop / timer / DNS resources. + */ +public interface PulsarHttpClientFactory { + PulsarHttpClient newHttpClient(PulsarHttpClientConfig config); +} + +/** + * ServiceLoader-discovered backend (AsyncHttpClient, JDK HttpClient, ...). + * Operators don't see this directly; the framework picks a provider once + * per PulsarClient and uses it to build every PulsarHttpClient instance. + */ +public interface PulsarHttpClientProvider { + String name(); // e.g. "asynchttpclient", "jdk" + default int priority() { return 0; } // higher wins; tie-broken by name + PulsarHttpClientFactory newFactory(PulsarHttpClientFactoryConfig sharedResources); +} +``` + +`HttpRequest` is an immutable value type with method, URI, headers, optional body (sealed `Body = Bytes | Form`), and optional timeout override. `HttpResponse` exposes status, headers, and the body bytes (buffered-only in v1; 16 MiB default cap configurable via `PulsarHttpClientConfig.maxResponseBodyBytes()`). `PulsarHttpClientConfig` carries per-instance concerns: a `ClientTlsPurposeContext` (the framework consults the configured `PulsarTlsMaterialProvider` for TLS material), default timeouts, proxy settings, the Pulsar user-agent string, default headers, and an optional `OpenTelemetryHooks` for tracing. The configuration deliberately does NOT carry the cert/key/trust material directly — that lives in the `PulsarTlsMaterialProvider` and is looked up by purpose. + +**Why multiple HTTP clients?** When the binary connection uses mTLS, the OAuth2 client may need a DIFFERENT mTLS configuration to talk to the identity provider — the IdP and the broker are different trust domains. Sharing one HTTP client instance across both is wrong. But the *resources* — event loop threads, DNS resolver cache, refresh scheduler — should be shared. The framework owns the resources via `PulsarHttpClientProvider.newFactory(sharedResources)` and hands out distinct `PulsarHttpClient` instances per `PulsarHttpClientConfig`. + +**Default implementation** — backed by AsyncHttpClient and the existing `PulsarHttpAsyncSslEngineFactory` bridge — ships in `pulsar-client-v5` (`name="asynchttpclient"`, `priority=100`) and is registered through `META-INF/services`. A JDK-`HttpClient`-backed provider can be shipped as an opt-in module for users who want zero AsyncHttpClient transitive dependency, with documented feature regressions (no SOCKS5, no custom DNS resolver, immutable `HttpClient` requires rebuild on TLS rotation). + +**Provider resolution**: when multiple providers are present, the v5 client picks by `builder.httpClientProvider(name)` if set, else by highest `priority()` (ties broken by `name()` ascending for determinism). `builder.httpClient(PulsarHttpClient)` overrides discovery entirely with a caller-supplied instance. + +Consumers of `PulsarHttpClient` after the v4 internal migration completes: + +- v5-native `AuthenticationOAuth2`, for token endpoint and well-known metadata fetches. +- v5-native `AuthenticationAthenz`, for ZTS interactions where applicable. +- v4-internal `HttpClient` (HTTP topic lookup) — migrated to use the v5 SPI under the hood. +- Any third-party plugin that needs HTTP. + +### Redesigned PIP-337 SSL provider: `PulsarTlsMaterialProvider` + +A small set of new types replaces the existing `PulsarSslFactory` / `PulsarSslConfiguration` surface as the v5 client's (and Pulsar 5.0's) TLS integration point. The SPI provides *material*; the framework builds and caches the `SSLEngine`/`SSLContext`. + +- **`PulsarTlsMaterialProvider`** (interface, in `org.apache.pulsar.common.tls`) — the v5 TLS SPI. The framework calls `getTlsMaterial(...)` with a `TlsPurposeContext` (described next) that identifies the TLS usage, and the provider returns the matching `TlsMaterial`; it also registers/unregisters `TlsMaterialListener`s and is initialized and closed by the framework. + + ```java + public interface PulsarTlsMaterialProvider { + CompletableFuture initialize(TlsMaterialProviderInitContext context); + /** Material for the context, applying the usage-identifier and purpose fallback chain when none is configured. */ + CompletableFuture getTlsMaterial(TlsPurposeContext purpose); + void registerListener(TlsPurposeContext purpose, TlsMaterialListener listener); + void unregisterListener(TlsPurposeContext purpose, TlsMaterialListener listener); + void close(); + } + ``` + Resolution first tries material registered for the context's specific usage identifier (when one is present), then the requested `purpose()`, then that purpose's fallback chain (e.g. `HTTP_LOOKUP → BINARY_CLIENT → GENERIC`); if nothing along the chain yields material, the request fails. + +- **`TlsPurposeContext`** (sealed interface, in `org.apache.pulsar.common.tls`) — identifies *why* TLS is requested and in what role. It is never instantiated directly: a context is always a `ClientTlsPurposeContext` or a `ServerTlsPurposeContext`. Sealing keeps the type evolvable — accessors or members can be added for the closed set of permitted implementations without breaking binary compatibility. This is shown below as sealed interfaces to express the model; the implementation may instead realize it as a value-class/record-based structure (e.g. final classes or records carrying the purpose, usage identifier, and server-side listener/host fields) if that proves simpler, as long as the same contract is preserved. The well-known purposes are captured as enums: + + ```java + package org.apache.pulsar.common.tls; + + public sealed interface TlsPurposeContext + permits ClientTlsPurposeContext, ServerTlsPurposeContext { + boolean isServer(); + + /** + * Optional finer-grained tag identifying a specific usage within a well-known + * purpose, so a provider can serve distinct material for that one usage (resolution + * otherwise falls back to the purpose). Example: the OAuth2 plugin's IdP HTTP client + * uses purpose GENERIC with usageIdentifier (AuthenticationOAuth2.class, "idpHttpClient"). + */ + Optional usageIdentifier(); + + /** Identifies a specific TLS usage: the owning class and a stable name within it. */ + record UsageIdentifier(Class owner, String identifier) {} + } + + /** Client-side purposes. Listener concerns never appear here — the client side stays simple. */ + public non-sealed interface ClientTlsPurposeContext extends TlsPurposeContext { + ClientPurpose purpose(); + @Override default boolean isServer() { return false; } + + /** + * Well-known client purposes. Each declares the purpose it FALLS BACK to when no + * material is configured for it; resolution walks the chain and fails if nothing is + * found. GENERIC has no fallback: it means "OS default trust store, no client + * certificate" (Netty's default when no custom trust store is set) — used for + * OAuth2 / IdP calls that must not reuse Pulsar-cluster TLS material. + */ + enum ClientPurpose { + GENERIC(null), + BINARY_CLIENT(GENERIC), // Pulsar binary protocol + HTTP_LOOKUP(BINARY_CLIENT), // HTTP topic lookup + ADMIN_HTTP(GENERIC); // admin HTTP client + final ClientPurpose fallback; + ClientPurpose(ClientPurpose fallback) { this.fallback = fallback; } + } + } + + /** Server-side purposes, carrying the broker-only listener/host context. */ + public non-sealed interface ServerTlsPurposeContext extends TlsPurposeContext { + ServerPurpose purpose(); + /** Advertised-listener name (e.g. "internal" / "external") when listener-specific. */ + Optional advertisedListenerName(); + /** Host the material is served for, when relevant. */ + Optional host(); + @Override default boolean isServer() { return true; } + + enum ServerPurpose { BROKER, PROXY, WEB_SERVICE } + } + ``` + + Keeping the listener/host context on `ServerTlsPurposeContext` only keeps it off the client path: the v4 client lets callers pass a listener name for binary/lookup *routing*, but the client's *TLS material* lookup never needs it, so client purposes stay a plain enum. On the broker side, different advertised listeners can carry different material — e.g. `internal` for in-cluster broker-to-broker traffic (possibly self-signed, possibly mTLS) and `external` for a publicly-trusted CA without mTLS. Broker-side advertised-listener-aware authentication/authorization is a separate future PIP. + + Code that needs to distinguish a specific usage within a well-known purpose attaches an optional **usage identifier** — a `(Class owner, String identifier)` record — to its `TlsPurposeContext`, instead of introducing a new purpose. The `purpose()` still returns a well-known enum that supplies the fallback chain; the usage identifier only lets a provider serve distinct material for that one usage when configured, and resolution otherwise falls back through the purpose's chain. For example, the OAuth2 plugin's IdP HTTP client uses `purpose() == GENERIC` with usage identifier `(AuthenticationOAuth2.class, "idpHttpClient")`: a deployment that pins a dedicated trust store for the IdP can register material against that identifier, while everything else resolves to the generic default. Replication and other broker-to-broker server purposes are added to `ServerPurpose` only when a concrete need appears. + +- **`TlsMaterial`** (interface, in `org.apache.pulsar.common.tls`) representing the TLS material and configuration for a given purpose. + - subinterfaces **`ClientTlsMaterial`** and **`ServerTlsMaterial`** + - default implementations `DefaultClientTlsMaterial` and `DefaultServerTlsMaterial` which are immutable value objects. + - `PulsarTlsMaterialProvider` implementations are responsible for handling updates efficiently. When an update is needed, the `getTlsMaterial` method should return a new `TlsMaterial` instance if the material has changed, or the same instance as was returned for the previous method call if it hasn't. This allows for efficient caching and avoids unnecessary updates in the implementation side. + - `TlsMaterial` will support both PEM file (pkcs#8) and keystore (pkcs12/jks) formats so that there won't be a need for separate configuration paths for different formats. Regardless of the source of the material, `TlsMaterial` will expose the following methods: + - `PrivateKey getPrivateKey()` + - `Iterable getKeyCertChain()` + - `Iterable getTrustCerts()` + - `Iterable getTlsProtocols()` + - `Iterable getTlsCiphers()` + - `ServerTlsMaterial` will also expose the following methods: + - `boolean isTrustedClientCertRequired()` (`false` by default, must be enabled for `mTLS` or TLS authentication.) + - `ClientTlsMaterial` will also expose the following methods: + - `boolean isHostnameVerificationRequired()` (`true` by default) + - `boolean isTrustAnyCaCert()` (`false` by default, matches the existing insecure mode when `true`. Breaks `mTLS` if set to `true`.) +- **`TlsMaterialListener`** (interface, in `org.apache.pulsar.common.tls`) representing a listener for TLS material updates. The provider implementation will first call the listener once the material has been loaded and is ready to be used. It will then call the listener again whenever the material changes. This is mainly useful for broker side TLS material updates since the broker will need to change the TLS configuration for the server when the material changes. On the client side, the implementation will request the TlsMaterial for each usage using the `getTlsMaterial` method. + +**Default file-based provider.** `FileBasedTlsMaterialProvider` implements `PulsarTlsMaterialProvider`, loading PEM and keystore files (supplied via a builder) and reloading them when their modification timestamps change. It delegates per-purpose lifecycle to `FileBasedClientTlsMaterialSource` (client) and `FileBasedServerTlsMaterialSource` (server), and supports **dynamic registration** of additional sources per purpose after construction: + +```java +/** Register an additional source for a purpose, augmenting any already registered for it. */ +void registerSource(TlsPurposeContext purpose, TlsMaterialSource source); + +/** Replace any source(s) registered for a purpose with the given one. */ +void setSource(TlsPurposeContext purpose, TlsMaterialSource source); +``` + +Registration is **augment-then-override per purpose**: a later `registerSource` for a purpose takes precedence over earlier ones for overlapping material, while `setSource` replaces outright. The split is needed for v4 API compatibility, where some TLS configuration comes from the client builder and some from a legacy `AuthenticationTls` plugin — the bridge augments rather than discarding a user-configured source. Registration is thread-safe (copy-on-write) relative to in-flight `getTlsMaterial` calls. These methods belong to the default `FileBasedTlsMaterialProvider`; a user who installs a fully custom `PulsarTlsMaterialProvider` is expected to configure its TLS material directly, and the v4 bridge then has nothing to register against. + +The v4 `AuthenticationTls` / `AuthenticationKeyStoreTls` bridge uses this to install a plugin's TLS material for the relevant client purposes without replacing a user-configured provider (in-scope item #7 and the [TLS override hook](#legacyv4authenticationadapter-v4--v5)). The broker's default `DefaultBrokerTlsMaterialProvider` is a thin `FileBasedTlsMaterialProvider` wrapper configured from the existing `ServiceConfiguration` properties. + +**Client configuration.** The default client-side `PulsarTlsMaterialProvider` is wired from the client builder. Rather than scattering many `tls*` setters across the builder, the TLS file/keystore settings live on a **`FileBasedTlsMaterialSource`**, created via its own builder and registered with the default `FileBasedTlsMaterialProvider` for one or more `TlsPurposeContext`s. The common case registers a single source shared across all client purposes; advanced deployments register distinct sources when trust domains differ (e.g. a dedicated source for `GENERIC` / OAuth2-IdP calls). Unconfigured purposes resolve through the fallback chain above. The exact builder method set is still being finalized (see [Open Questions](#open-questions)); the goal is to keep ordinary use simple while still covering advanced enterprise cases (mTLS to an OAuth IdP, etc.). + +**Custom providers** implement `PulsarTlsMaterialProvider` directly and may externalize the entire TLS configuration (e.g. to a KMS). A custom provider that still uses files can reuse `FileBasedClientTlsMaterialSource` / `FileBasedServerTlsMaterialSource` for rotation. The same implementation can serve both client and broker sides, or differ between them (a server-side KMS integration often differs from the client-side one). + +**Removal.** The existing v4 `org.apache.pulsar.common.util.PulsarSslFactory` and `PulsarSslConfiguration` are removed completely — an intentional breaking change. Because the SPI now provides only material and the framework owns the `SSLEngine`/`SSLContext` (built and cached, rebuilt only when `TlsMaterial` equality changes), the `org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext` class is no longer needed either; the file-based provider reads keystores (PKCS12/JKS) directly for the private key, key-cert chain, and trust certificates. PIP-337 is not broadly used, so it is removed rather than retained alongside the new SPI. + +### `LegacyV4AuthenticationAdapter` (v4 → v5) + +Wraps an arbitrary v4 `Authentication` instance as a v5 `Authentication`. Detects the plugin's style by inspecting which `AuthenticationDataProvider.has*()` methods return true, and declares the matching v5 capability: + +```java +package org.apache.pulsar.client.impl.v5.auth; + +public final class LegacyV4AuthenticationAdapter implements Authentication { + + private final org.apache.pulsar.client.api.Authentication v4; + private AuthenticationInitContext ctx; + + /** + * Wraps {@code v4} and returns an instance that declares the right + * capability interfaces for its style — typically + * {@link BinaryProtocolAuthDataProvider} (+ {@link HttpAuthHeadersProvider}) + * for one-pass plugins (Token, Basic, OAuth2, Athenz) or + * {@link ChallengeResponseHandler} for SASL. + * Plugins reporting {@code hasDataForTls() == true} have their TLS material + * registered with the client's {@link PulsarTlsMaterialProvider} by the bridge + * and are represented by the built-in {@link TlsAuthentication} plugin (see "TLS + * override hook" below). + */ + public static Authentication wrap(org.apache.pulsar.client.api.Authentication v4); + + @Override public void configure(Map p) { v4.configure(p); } + @Override public CompletableFuture initializeAsync(AuthenticationInitContext c); + @Override public void close() throws Exception; +} +``` + +Internally, the concrete subclasses cover the cases: + +- `LegacyV4CredentialAdapter implements BinaryProtocolAuthDataProvider, HttpAuthHeadersProvider` — for plugins reporting `hasDataFromCommand()` or `hasDataForHttp()`. Renders the credential into both transport forms (binary bytes = `v4.getAuthData(host).getCommandData()`; HTTP headers from `v4.getAuthData(host).getHttpHeaders()`). Adds the v4 `Authentication.authenticationStage(...)` HTTP hook for stages that need a callback. +- `LegacyV4ChallengeResponseAdapter implements BinaryProtocolAuthDataProvider, ChallengeResponseHandler` — for plugins implementing the v4 `authenticate(AuthData)` challenge-response method. Produces the initial frame via `getAuthDataAsync(...)` and routes binary challenges through `respondToChallengeAsync(...)`; HTTP challenges unsupported (matches v4 behaviour). +- `LegacyV4TlsAdapter extends TlsAuthentication` — for plugins reporting `hasDataForTls() == true` (v4 `AuthenticationTls` / `AuthenticationKeyStoreTls`, or third-party equivalents). It reuses the built-in `TlsAuthentication` plugin (so the binary protocol sends `auth_method_name=tls`), and the bridge registers the plugin's TLS material with the client's `PulsarTlsMaterialProvider` — see the TLS override hook below. + +Three invariants apply across all adapters: + +1. **Always offload to `ctx.scheduler()`.** Even calls to "fast" v4 implementations like `AuthenticationToken.getAuthData()` go through the scheduler. The cost is a thread hop, but the simplicity is worth it: the adapter has no class-name allow-list, no per-impl heuristics, and no production hazard from a misclassified plugin. +2. **Translate v4 exceptions.** Any `org.apache.pulsar.client.api.PulsarClientException` thrown by the v4 call is wrapped in the v5 `org.apache.pulsar.client.api.v5.PulsarClientException` and used to complete the returned future exceptionally. +3. **Refresh is the plugin's concern.** The framework has no built-in refresh; proactive renewal and token reuse are internal to the authentication implementation. A v5-native plugin (e.g. `AuthenticationOAuth2`, `AuthenticationAthenz`) renews its short-lived credential internally and returns the current one whenever the framework next calls its async credential method — and the broker's `CommandAuthChallenge` refresh sentinel simply triggers that same call. The legacy adapter therefore just re-invokes the wrapped v4 plugin per request; no refresh metadata crosses the capability surface. + +**TLS override hook.** When the v5 client builder is asked to use a v4 plugin that reports `hasDataForTls() == true` (notably the v4 `AuthenticationTls` and `AuthenticationKeyStoreTls` classes, or any third-party equivalent), the bridge does two things during client construction: (1) it **registers** the plugin's TLS material with the client's default `FileBasedTlsMaterialProvider` for the relevant client purpose-contexts — via the provider's dynamic source-registration API (see [`PulsarTlsMaterialProvider`](#redesigned-pip-337-ssl-provider-pulsartlsmaterialprovider)) rather than replacing a provider the user may have configured — and (2) it represents the connection's authentication with the built-in `TlsAuthentication` plugin so the binary protocol sends `auth_method_name=tls`. The `LegacyV4AuthenticationAdapter` exposes a package-private `extractTlsMaterial()` that the builder calls to obtain the cert/key/trust sources for registration. + +This preserves v4's "`AuthenticationTls` is an `Authentication` that overrides TLS" semantics for source compatibility, now expressed cleanly as builder-level TLS material **plus** the built-in `TlsAuthentication` plugin. + +### `ClientCnx` async-driver carve-out (generic, not SASL-specific) + +The v5 client wraps the v4 transport (per PIP-466), so the practical change is twofold: + +1. `V5ToV4AuthenticationAdapter` exposes the v4 `Authentication` interface that `ClientCnx` already drives. +2. A new public interface `org.apache.pulsar.client.api.internal.AsyncAuthenticationDriver` (in `pulsar-client-api`) lets `ClientCnx` detect when the wrapped authentication supports an async path: + +```java +package org.apache.pulsar.client.api.internal; + +public interface AsyncAuthenticationDriver { + CompletableFuture getAuthDataAsync(String brokerHostName); + CompletableFuture authenticateAsync(AuthData challenge, String brokerHostName); +} +``` + +The `.internal.` subpackage signals "stable internal — application code should not implement this." `ClientCnx.newConnectCommand()` and `ClientCnx.handleAuthChallenge(...)` check `authentication instanceof AsyncAuthenticationDriver async` and route through the async API when present, piping the result back through `whenCompleteAsync(..., ctx.executor())` for the Netty thread. **Generic across all challenge types** — connect, REFRESH, SASL multi-round, custom challenge-response. ~50-60 LOC total in `ClientCnx`. The marker is the only carve-out from the otherwise-untouched v4 client. + +When `authentication` is a plain v4 instance (not `AsyncAuthenticationDriver`), `ClientCnx` preserves the existing sync path verbatim — no behavioral change for v4-only callers. + +After the v4 internal migration (in-scope item #7) completes, the built-in v4 classes themselves wrap `V5ToV4AuthenticationAdapter` over their v5-native bodies, so even users of v4 `PulsarClient` get the async path automatically when they use a built-in plugin. On this v4-facing path, `V5ToV4AuthenticationAdapter` maps v5 auth exceptions **back** to the corresponding v4 `org.apache.pulsar.client.api.PulsarClientException` subtypes — v4 exception types are part of the public API and must be preserved for v4 callers (see the Error model). + +### HTTP multi-round auth drivers + +The binary protocol runs its multi-round loop in one place (`ClientCnx`, above). HTTP instead has two client stacks — the JAX-RS admin client and the AsyncHttpClient HTTP-lookup client — so the `401`→resubmit→`200` loop is implemented by a framework-side **driver** per stack rather than inside the plugin (the v4 hazard the [Transport B](#transport-b--httphttps-rest-api-admin-client-http-topic-lookup) note describes). A driver detects the challenge style from the capability the plugin declares — today only the SASL-style `HttpHeaderChallengeResponseHandler` — surfaces each server challenge through `HttpAuthCallContext.serverChallengeHeaders()`, and attaches the plugin's computed headers to the resubmitted request. The SASL driver reproduces v4 behaviour exactly, including re-issuing the exchange as a `GET` to the original URI. The precise driver/interface factoring (and a future standard `WWW-Authenticate` driver) is still being settled — see [Open Questions](#open-questions). + +### Class-name compatibility and the v4 internal migration + +`authPluginClassName` strings in `ClientConfigurationData` continue to work without modification: + +- The existing v4 class names (`org.apache.pulsar.client.impl.auth.AuthenticationToken`, `AuthenticationTls`, `AuthenticationOAuth2`, `AuthenticationAthenz`, `AuthenticationBasic`, `AuthenticationSasl`, `AuthenticationDisabled`, plus the v4 `AuthenticationKeyStoreTls`) are retained. +- **After the v4 internal migration (in-scope item #7)**, each of those classes is a thin shim that internally constructs a v5-native implementation and forwards the v4 `Authentication` calls to it. The shim's body becomes mostly: + ```java + public class AuthenticationToken implements org.apache.pulsar.client.api.Authentication { + private final transient TokenAuthenticationV5 delegate = new TokenAuthenticationV5(); + @Override public String getAuthMethodName() { return delegate.authMethodName(); } + @Override public void configure(Map params) { delegate.configure(params); } + @Override public void start() throws PulsarClientException { /* delegate.initializeAsync().get() */ } + @Override public AuthenticationDataProvider getAuthData(String host) { + // Synthesised from delegate.getAuthDataAsync(...) — cached, hostname keying + // preserved at this layer for source compatibility. + } + // ... rest of v4 surface routed through delegate ... + } + ``` +- The v4 `AuthenticationTls` / `AuthenticationKeyStoreTls` are also shims, but rather than wrapping a v5 auth they build file-based TLS material sources from the configured paths/keystore, register them with the client's `FileBasedTlsMaterialProvider` for the relevant client purposes (via the dynamic source-registration API), and use the built-in `TlsAuthentication` plugin so the binary protocol sends `auth_method_name=tls`. +- Third-party class names (or v4 classes without a v5-native equivalent) go through `LegacyV4AuthenticationAdapter` unchanged. +- `AuthenticationFactory.create(authPluginClassName, authParamsString)` and `AuthenticationFactory.create(authPluginClassName, Map)` continue to instantiate via `AuthenticationUtil.create()`, unchanged. + +The two configuration paths (programmatic vs string-based) both work for the v4 surface: + +- **Programmatic**: `new AuthenticationToken(jwt)` — the shim's constructor stores the JWT, configures the underlying `TokenAuthenticationV5` directly. v4's `start()` triggers `initializeAsync(...).get()`. +- **String-based**: `AuthenticationUtil.create("...AuthenticationToken", "{...}")` — reflection invokes the no-arg constructor, then `configure(params)` is called, which the shim forwards to `delegate.configure(params)`. v4's `start()` triggers `initializeAsync(...).get()`. + +### Error model + +All async failures complete the returned `CompletableFuture` exceptionally with a v5 `org.apache.pulsar.client.api.v5.PulsarClientException`. The relevant subclasses for auth are: + +- `AuthenticationException` — terminal authentication failure (rejected credential, untrusted certificate). Connection is failed. +- `GettingAuthenticationDataException` — transient failure during credential acquisition (token endpoint timeout, ZTS unavailable). The caller may retry; the v5 connection layer treats this the same way it treats network errors. +- `UnsupportedAuthenticationException` — the requested capability is not supported by the wrapped impl (used by `LegacyV4AuthenticationAdapter` when a v4 provider returns null/false from the corresponding `hasData*` method). + +Translation runs in **both** directions, since v4 and v5 exception types are both public API: + +- **v4 → v5** — `LegacyV4AuthenticationAdapter` wraps any v4 `org.apache.pulsar.client.api.PulsarClientException` thrown by a wrapped v4 plugin onto the v5 subclasses above. +- **v5 → v4** — on the v4-facing path (`V5ToV4AuthenticationAdapter` and the built-in v4 shims), v5 auth exceptions are mapped back to the matching v4 `PulsarClientException` subtypes so v4 callers see the exceptions they always have. This is a straightforward one-to-one mapping (v5 `AuthenticationException` → v4 `AuthenticationException`, and so on) that introduces no behavioural change; the acceptance criterion is that the existing v4 authentication tests pass unmodified after the migration. + +## Public-facing Changes + +### Public API + +New public types introduced by this PIP: + +- `org.apache.pulsar.client.api.v5.auth` — `Authentication` (replacing the PIP-466 sync stub); capability interfaces `BinaryProtocolAuthDataProvider`, `HttpAuthHeadersProvider`, `ChallengeResponseHandler`, `HttpHeaderChallengeResponseHandler`; convenience composites `SinglePassAuthentication`, `ChallengeResponseAuthentication`; contexts `AuthenticationInitContext`, `StatefulCallContext`, `AuthenticationCallContext`, `HttpAuthCallContext`; value types `BinaryProtocolAuthData`, `DefaultBinaryProtocolAuthData`, `HttpAuthHeaders`, `AuthChallenge`, `ChallengeResponse`; exceptions `AuthenticationException`, `GettingAuthenticationDataException`, `UnsupportedAuthenticationException`. +- `org.apache.pulsar.client.api.v5.http` — `PulsarHttpClient`, `PulsarHttpClientFactory`, `PulsarHttpClientProvider`, `PulsarHttpClientConfig`, `HttpRequest`, `HttpResponse`. +- `org.apache.pulsar.common.tls` — `PulsarTlsMaterialProvider`, `TlsMaterialProviderInitContext`; purpose contexts `TlsPurposeContext` (with the nested `UsageIdentifier` record), `ClientTlsPurposeContext`, `ServerTlsPurposeContext`; `TlsMaterial` / `ClientTlsMaterial` / `ServerTlsMaterial`, `DefaultClientTlsMaterial` / `DefaultServerTlsMaterial`; `TlsMaterialListener`; `FileBasedTlsMaterialProvider`, `DefaultBrokerTlsMaterialProvider`, `TlsMaterialSource`, `FileBasedClientTlsMaterialSource`, `FileBasedServerTlsMaterialSource`. +- `org.apache.pulsar.client.api.internal` — `AsyncAuthenticationDriver` (observed by `ClientCnx`; application code should not implement it). +- `pulsar-client-v5` (`org.apache.pulsar.client.impl.v5.auth`) — the built-in `TlsAuthentication` mTLS plugin, and the compatibility adapters `LegacyV4AuthenticationAdapter` and `V5ToV4AuthenticationAdapter`. + +Removed: `org.apache.pulsar.common.util.PulsarSslFactory`, `PulsarSslConfiguration`, and `org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext` (intentional breaking change — see Backward & Forward Compatibility). + +### Binary protocol + +No new wire-protocol commands. The existing `CommandConnect`, `CommandAuthChallenge`, and `CommandAuthResponse` are used unchanged. + +### Configuration + +No new broker configuration. The v5 client builder gains methods to select/override the HTTP client (`httpClient(PulsarHttpClient)`, `httpClientProvider(String name)`), supply a custom `PulsarTlsMaterialProvider`, and register one or more `FileBasedTlsMaterialSource`s — each built via its own builder from cert/key/trust paths or a keystore, ciphers/protocols, and hostname-verification / insecure-connection flags — for the desired `TlsPurposeContext`s. The exact builder surface is being finalized — see [Open Questions](#open-questions). + +### CLI + +No new CLI commands. + +# Open Questions + +The following design decisions remain open, each with a suggested direction. + +1. **HTTP multi-round "driver" design and the SASL-vs-standard interface split.** v5 moves the HTTP multi-round loop out of the plugin and into framework-side drivers (one for the JAX-RS admin client, one for the AsyncHttpClient HTTP-lookup client), and anticipates two HTTP challenge-response capability interfaces — the SASL-style `HttpHeaderChallengeResponseHandler` (shipping first, preserving the existing custom-header and `GET`-to-original-URI behaviour) and a future standard `WWW-Authenticate` / digest interface. Still undecided: the exact interface names and method shapes, how a driver selects between styles, and how much of the `401`→resubmit→`200` state machine is shared across the two HTTP stacks. *Suggested:* keep `HttpHeaderChallengeResponseHandler` as the SASL-style interface; add a sibling (e.g. `StandardHttpChallengeResponseHandler`) only when a standard-scheme plugin is actually implemented; have each driver dispatch by `instanceof` on the capability the plugin declares; and share one `401`→resubmit→`200` state machine behind a thin per-stack request/response adapter so the JAX-RS and AsyncHttpClient drivers don't duplicate the loop. + +2. **Exact v5 client builder surface for TLS configuration and HTTP-client selection.** The underlying model is already settled earlier in this document: the default file-based provider is wired from the client builder via `FileBasedTlsMaterialSource`s registered per `TlsPurposeContext`. The open question is the builder ergonomics — keeping the **common single-source case** convenient while still exposing the **fine-grained per-purpose** path — together with the exact method names and overloads and how HTTP-client selection (`httpClient(...)` / `httpClientProvider(...)`) sits alongside. To be finalized in a later revision of this PIP. + +# Security Considerations + +- **TLS verification defaults.** The default `PulsarHttpClient` implementation enables TLS verification and hostname checking by default. `tlsAllowInsecureConnection` and `tlsHostnameVerificationEnable` from `ClientConfigurationData` are honored for parity with v4, but enabling an insecure setting is logged once at WARN level, on first use, to surface insecure deployments without flooding the log on every request. + +# Backward & Forward Compatibility + +- The v5 client is published as a standalone artifact in 5.0.0-M1 (the existing `pulsar-client-v5` and `pulsar-client-api-v5` modules from PIP-466); folding the v5 client into `pulsar-client-shaded` is acknowledged as a follow-up gap and tracked separately. +- PIP-337 is removed from Pulsar in Pulsar 5.0 since retaining it in the code base causes an additional maintenance burden and most users don't use it. Existing PIP-337 users will need to migrate to the new `PulsarTlsMaterialProvider` SPI. + +## Upgrade + +- Existing applications using `pulsar-client-api` (v4) require **no changes**. The v4 surface is untouched; the deprecated v4 methods (`getAuthData()` no-arg, `configure(Map)`) are retained per PIP-466's stability promise. The only v4 module addition is a new public interface `org.apache.pulsar.client.api.internal.AsyncAuthenticationDriver` (in a `.internal.` subpackage) — application code should not implement it; it is observed by `ClientCnx` to opt into the async path. +- Applications using `pulsar-client-api-v5` (v5) gain the new SPI. Existing `authPluginClassName` strings continue to work across both the v4 and v5 client APIs. +- Mixed v4 + v5 Client API usage in the same JVM is supported. +- The `Serializable` contract on the v4 `Authentication` interface remains, supporting Pulsar Functions and connector frameworks that serialize auth instances. The v5 `Authentication` interface deliberately does **not** extend `Serializable`; `V5ToV4AuthenticationAdapter.writeObject` throws `NotSerializableException` with an actionable message pointing to the `authPluginClassName` + `authParams` migration path. Connectors that serialize auth across class loaders must continue to use the v4 interface or the configuration-based approach. This is the stance for now; specific configuration-serialization needs for frameworks such as Apache Flink will be addressed when Flink support is added to Pulsar 5.0. + +## Downgrade / Rollback + +- Removing the v5 SPI types is a source-incompat break for any application that compiled against them, but the v4 surface is always a rollback target — applications can pin to the v4 `pulsar-client` and the new types are simply unused. + +## Pulsar Geo-Replication Upgrade & Downgrade/Rollback Considerations + +Authentication is a per-client concern. Geo-replication does not require any auth-related changes. Each cluster authenticates the replicator client independently using whichever auth plugin is configured. + +# Alternatives + +- **Pure additive `AuthenticationAsync` alongside v4.** Add async methods to a new sibling interface but keep the v4 surface as the data model. Rejected: it extends the kitchen-sink `AuthenticationDataProvider` problem rather than fixing it. Implementations would still mix TLS, HTTP, and command-data concerns in one type. + +- **Reusing AsyncHttpClient directly without an SPI.** Skip `PulsarHttpClient` and just expose AsyncHttpClient on `AuthenticationInitContext`. Rejected: ties Pulsar's public API to the AsyncHttpClient transitive dependency, blocks future migration to JDK HttpClient or other libraries, and gives users no way to inject custom HTTP behavior cleanly. + +- **Splitting the PIP-337 cleanup into a separate sibling PIP.** Rejected. Folding the relocation into this PIP keeps the auth-to-TLS decoupling in one document, avoids interleaved release ordering between two related PIPs, and gives reviewers a single place to evaluate the overall design. + +- **`@Deprecated` and remove the v4 `Authentication` interface outright.** Rejected: PIP-466 explicitly committed v4 to remain unchanged, and the v4 `Authentication` interface carries `@InterfaceStability.Stable`. Removing it would break v4 API usage and every third-party auth plugin in Pulsar 5.0. + +# Links + +- [PIP-97: Asynchronous Authentication Provider](pip-97.md) +- [PIP-337: SSL Factory Plugin](pip-337.md) +- [PIP-466: New Java Client API (V5)](pip-466.md) +- [PIP-460: Scalable Topics](pip-460.md) +- [GitHub issue 24795 (HTTP client pluggability)](https://github.com/apache/pulsar/issues/24795) +- [GitHub PR 24944 (oauth2 trustcerts file and timeouts)](https://github.com/apache/pulsar/pull/24944) +- Mailing List discussion thread: https://lists.apache.org/thread/s9n9jksr9vqgn9o982zmnnkcxdcncy3f +- Mailing List voting thread: