[Bug] S3ClientFactory caches one S3Client per bucket forever with no eviction — unbounded native memory growth with per-bucket catalogs (e.g. S3 Tables)

[ktopcuoglu]

What's Wrong?

S3ClientFactory::_cache (be/src/util/s3_util.h) is a plain std::unordered_map<S3ClientConf, ..., S3ClientConfHash> with no eviction, TTL, or size cap:

auto it = _cache.find(s3_conf);
if (it != _cache.end()) { ... }
...
auto [it, _] = _cache.emplace(s3_conf, std::move(obj_client));

(s3_util.cpp, S3ClientFactory::create — identical on master and branch-4.1/4.1.4.)

S3ClientConf's identity includes the bucket, so every distinct bucket a BE process ever accesses creates and permanently retains a new cached client. This is a real problem for any catalog where storage is bucket-per-table rather than bucket-per-warehouse — e.g. AWS S3 Tables, which auto-provisions one bucket per Iceberg table. A long-lived BE process that scans N distinct tables accumulates N permanently-cached Aws::S3::S3Client objects, each with its own libcurl connection pool (CA bundle parsed per pooled handle) and, when using STS/WebIdentity-style credentials, an auxiliary STS credentials-provider sub-client built via a bare default constructor — bypassing Doris's own caFile/connection settings entirely.

Confirmed via jemalloc heap profiling (prof_active) that this is genuinely live allocated memory (jemalloc allocated, not just retained/fragmented resident), invisible to Doris's own MemTracker/workload-group accounting — it shows up purely as UntrackedMemory while workload groups report ~0 the whole time. One measured instance attributed roughly 700 MiB to these allocation sites after a routine ETL window, in a steadily-climbing, never-plateauing pattern that eventually leads to the process being killed under memory pressure.

How to Reproduce

1. Configure an external catalog (e.g. Iceberg REST) where table storage maps to a distinct S3 bucket per table (S3 Tables does this automatically).
2. From a single long-lived BE process, scan many distinct tables over time, no restarts between them.
3. Observe RSS / jemalloc allocated climbing roughly linearly with the number of distinct buckets ever touched, never plateauing, independent of query concurrency or workload-group memory settings.
4. Workload-group memory metrics stay near zero throughout — the growth is invisible from Doris's own accounting.

What You Expected

Some bound on S3ClientFactory's cache — idle-TTL and/or LRU eviction, similar to the bounded/expiring cache (_azure_cache, capacity 256, LRU + expiry pruning) added for Azure clients in #68117. That PR explicitly left the non-Azure _cache untouched.

Version

Reproduced on the official 4.1.4 release (branch-4.1). Same unbounded _cache design confirmed present on master by source inspection.

Anything Else?

Happy to contribute a patch — fix direction: capacity-bounded LRU + idle-TTL eviction on _cache (mirroring #68117's _azure_cache); share one credentials-provider/STS client across cache entries that share the same credential config instead of one per bucket; and separately, the maxConnections fallback to 102400 in _create_s3_client when no value is supplied looks worth revisiting too.

Related but distinct: #66300 (different trigger, same "unbounded S3 connection" smell); #66997/#65416/#67056 (separate BE thread-pool leak, unrelated to this cache).

• Yes, willing to submit a PR.

---

This issue was drafted by an AI coding assistant (Claude) based on source-code review and a heap-profiling investigation carried out by our engineering team, and was reviewed and submitted by a human maintainer of this report.

[Doris-Breakwater]

Initial maintainer analysis

Assessment: confirmed BE client-cache lifetime/design bug; keep this issue open. The issue currently has no labels or assignee. I recommend triaging it as an object-storage/BE resource-lifetime bug and considering a 4.1 backport once the fix is proven.

Verified facts

• On the 4.1.4 tag, S3ClientConf full-field equality and its hash include bucket, while URI conversion always copies the URI bucket into client_conf.bucket (source). S3ClientFactory::create() only looks up and inserts into the process-wide singleton's _cache; there is no erase, capacity, TTL, or lifecycle reset. Therefore every distinct full configuration is retained until BE shutdown, and otherwise-identical configurations with different buckets necessarily create distinct entries.
• Current master (573c93c63b88fb68ac1d318dcd3e1672dd2361e6) and current branch-4.1 (5e1ee245e23adedc0e9fff8b6006f067653e9155) retain the same behavior. This is not limited to the reported release.
• Each miss constructs a new Aws::S3::S3Client and credentials provider. Importantly, _create_s3_client() does not consume s3_conf.bucket; bucket is supplied on each object request. The bucket is construction identity for Azure's container client, but not for the non-Azure S3-compatible client. This makes bucket-sensitive identity avoidable for the reported non-Azure case.
• The code proves monotonically growing live ownership of client objects as new cache identities are seen. It does not, by itself, prove the reported ~700 MiB magnitude or the exact libcurl/OpenSSL/CA allocation breakdown. Those measurements are plausible and valuable reporter evidence, but the underlying heap-profile artifacts are still needed to independently verify per-entry cost and the dominant allocation stacks.

Scope corrections

• [feature](be) Native Azure credentials, SAS expiry and client lifetime in S3ClientFactory #68117 is currently an open PR, not behavior already present in master or 4.1. Its proposed Azure-only cache is a useful implementation reference (capacity-bounded LRU plus credential-expiry pruning), but it should not be described as an already-landed Doris cache.
• The maxConnections = 102400 fallback is real, but it is a ceiling, not an eager allocation of 102400 connections. It may amplify per-client transport use under concurrency, but it is separate from the cache-cardinality/lifetime defect and should be handled independently.
• The credentials-provider statement should be narrowed. In 4.1.4, the v2 WebIdentity path does default-construct STSAssumeRoleWebIdentityCredentialsProvider, while explicit role_arn AssumeRole paths construct an STS client with Doris's selected CA configuration. Current master likewise passes sts_client_config to explicit AssumeRole, but still default-constructs the WebIdentity base provider. Thus the bypass concern applies to the WebIdentity-default path, not to every STS-style provider.

Recommended fix direction

1. Introduce a provider-specific cache key. For non-Azure S3-compatible clients, omit bucket; retain it for Azure/container-bound clients. Do this in a cache-key type or normalization step rather than changing global S3ClientConf::operator==, which is also used by ObjClientHolder reset logic. This directly collapses bucket-per-table workloads to one client when transport and credential configuration are otherwise identical.
2. Bound the remaining non-Azure configuration space with capacity plus idle TTL/LRU eviction. Eviction should drop only the cache's shared_ptr, so active readers remain valid. Preserve construction outside the mutex, double-check on publication, and avoid destroying an evicted/losing AWS client while holding the factory lock.
3. Add low-cardinality observability: current entries, hits, misses, evictions, and client creations by provider. This will make the reported bucket-count/memory correlation testable in production.
4. Reassess a separate credentials-provider cache only after key normalization. Reusing one S3 client across buckets already reuses its provider; sharing providers across genuinely different/rotating token configurations has expiry and refresh semantics that need dedicated tests.

Suggested tests: two non-Azure buckets with identical transport/credentials reuse one fake client; Azure buckets remain distinct; capacity and idle expiry evict the expected entry while an externally held shared_ptr remains usable; credential/token changes still create a new identity; concurrent misses publish one retained entry; and destructor counters prove evicted clients are released.

Evidence requested to quantify impact and validate the regression

Please attach, with credentials/endpoints/bucket names redacted:

1. Before/after jemalloc heap dumps (or jeprof/flamegraph outputs) for a known number of newly touched buckets, including resolved cumulative stacks for S3 client, curl, TLS/CA, and credential-provider allocations.
2. A synchronized time series of distinct buckets touched, create one s3 client log count, jemalloc allocated, RSS, and Doris UntrackedMemory; include a control that repeatedly scans already-seen buckets and the drop after BE restart.
3. The effective credential configuration (aws_credentials_provider_version, provider type, whether role_arn is set, and whether AK/SK/token values rotate per table) and effective max_connections, without secret values. In particular, confirm whether the per-table configurations differ only by bucket or also by temporary credentials.
4. The number of buckets/clients behind the reported ~700 MiB delta and the approximate bytes-per-new-client slope.

These artifacts are not required to establish the unbounded-ownership bug, but they are needed to validate the claimed allocation sites, choose a safe default bound/TTL, and create a meaningful memory regression test.

Breakwater-GitHub-Analysis-Slot: slot_2220f6d6b149