Skip to content

Expose disposeInactivePoolsInBackground configuration for HttpClient ConnectionProvider #4165

@sugerboy20170827

Description

@sugerboy20170827

Problem Description

Spring Cloud Gateway (SCG) exposes HttpClientProperties to configure the underlying Reactor Netty connection pool, but it does not expose the disposeInactivePoolsInBackground(interval, inactivity) parameter from ConnectionProvider.Builder.

This causes a critical memory leak in environments with dynamic downstream instances (e.g., Kubernetes), where Pod IPs change frequently due to rolling deployments or auto-scaling.

Root Cause Analysis

Reactor Netty's PooledConnectionProvider maintains a ConcurrentHashMap<PoolKey, InstrumentedPool> named channelPools. A PoolKey is composed of:

  • remoteAddress (downstream IP:Port)
  • pipelineKey (channelHash, derived from HttpClientConfig)

Every time a downstream Pod is restarted or replaced, a new IP is assigned → a new PoolKey is created → a new SimpleDequePool is added to channelPools.

The critical problem: channelPools is never cleaned up automatically. The existing configuration options only clean up connections within a pool, not the PoolKey entries (pools themselves):

Config property Effect
httpclient.pool.max-idle-time Evicts idle connections inside a pool ✅
httpclient.pool.max-life-time Evicts long-lived connections inside a pool ✅
httpclient.pool.eviction-interval Triggers periodic connection eviction ✅
disposeInactivePoolsInBackground Removes PoolKey entries for inactive pools ❌ Not exposed

The disposeInactivePoolsInBackground method exists in PooledConnectionProvider (reactor-netty v1.1.x):

final void disposeInactivePoolsInBackground() {
    toDispose = channelPools.entrySet()
        .stream()
        .filter(p -> p.getValue().metrics().isInactiveForMoreThan(poolInactivity))
        .collect(Collectors.toList());

    toDispose.forEach(e -> {
        if (channelPools.remove(e.getKey(), e.getValue())) { // Actually removes the PoolKey!
            e.getValue().dispose();
        }
    });
    scheduleInactivePoolsDisposal();
}

This is triggered only when inactivePoolDisposeInterval is non-zero, which is set via ConnectionProvider.Builder.disposeInactivePoolsInBackground(interval, inactivity). SCG never sets this value, so it defaults to Duration.ZERO and the cleanup never runs.

Observed Impact

In our Kubernetes-based Spring Cloud Gateway deployment, we observed:

  • 541,106 instances of reactor.netty.internal.shaded.reactor.pool.SimpleDequePool
  • Occupying ~14.3 GB (94.34% of heap)
  • GC Root path: HttpResources → DefaultPooledConnectionProvider → channelPools (ConcurrentHashMap) → SimpleDequePool
  • Each PoolKey.fqdn corresponds to a distinct downstream Pod IP that no longer exists

Heap dump analysis confirmed channelPools.size() equals the number of accumulated SimpleDequePool instances, directly corresponding to the number of distinct downstream IPs seen over the application's lifetime.

Current Workaround

Users must bypass HttpClientProperties entirely and register a custom ConnectionProvider bean:

@Bean
@Primary
public ConnectionProvider gatewayConnectionProvider() {
    return ConnectionProvider.builder("gateway")
        .maxConnections(500)
        .maxIdleTime(Duration.ofSeconds(4))
        .maxLifeTime(Duration.ofSeconds(60))
        .evictInBackground(Duration.ofSeconds(10))
        // Not available via application.properties!
        .disposeInactivePoolsInBackground(
            Duration.ofMinutes(5),  // scan interval
            Duration.ofMinutes(5)   // inactivity threshold
        )
        .build();
}

This workaround works but requires users to know the internal reactor-netty API and loses the convenience of application.properties configuration.

Feature Request

Please expose disposeInactivePoolsInBackground in HttpClientProperties (or GatewayProperties) to allow configuration via:

spring.cloud.gateway.httpclient.pool.inactive-pool-dispose-interval=5m
spring.cloud.gateway.httpclient.pool.pool-inactivity=5m

And wire it into HttpClientAutoConfiguration when building the ConnectionProvider:

// In HttpClientAutoConfiguration or similar
if (pool.getInactivePoolDisposeInterval() != null) {
    builder.disposeInactivePoolsInBackground(
        pool.getInactivePoolDisposeInterval(),
        pool.getPoolInactivity()
    );
}

Environment

  • Spring Cloud Gateway: 3.x (based on Spring Boot 3 / SCG 4.x series)
  • Reactor Netty: 1.1.5
  • Deployment: Kubernetes with frequent Pod rolling updates

References

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions