Problem Description
Spring Cloud Gateway (SCG) exposes HttpClientProperties to configure the underlying Reactor Netty connection pool, but it does not expose the disposeInactivePoolsInBackground(interval, inactivity) parameter from ConnectionProvider.Builder.
This causes a critical memory leak in environments with dynamic downstream instances (e.g., Kubernetes), where Pod IPs change frequently due to rolling deployments or auto-scaling.
Root Cause Analysis
Reactor Netty's PooledConnectionProvider maintains a ConcurrentHashMap<PoolKey, InstrumentedPool> named channelPools. A PoolKey is composed of:
remoteAddress (downstream IP:Port)
pipelineKey (channelHash, derived from HttpClientConfig)
Every time a downstream Pod is restarted or replaced, a new IP is assigned → a new PoolKey is created → a new SimpleDequePool is added to channelPools.
The critical problem: channelPools is never cleaned up automatically. The existing configuration options only clean up connections within a pool, not the PoolKey entries (pools themselves):
| Config property |
Effect |
httpclient.pool.max-idle-time |
Evicts idle connections inside a pool ✅ |
httpclient.pool.max-life-time |
Evicts long-lived connections inside a pool ✅ |
httpclient.pool.eviction-interval |
Triggers periodic connection eviction ✅ |
disposeInactivePoolsInBackground |
Removes PoolKey entries for inactive pools ❌ Not exposed |
The disposeInactivePoolsInBackground method exists in PooledConnectionProvider (reactor-netty v1.1.x):
final void disposeInactivePoolsInBackground() {
toDispose = channelPools.entrySet()
.stream()
.filter(p -> p.getValue().metrics().isInactiveForMoreThan(poolInactivity))
.collect(Collectors.toList());
toDispose.forEach(e -> {
if (channelPools.remove(e.getKey(), e.getValue())) { // Actually removes the PoolKey!
e.getValue().dispose();
}
});
scheduleInactivePoolsDisposal();
}
This is triggered only when inactivePoolDisposeInterval is non-zero, which is set via ConnectionProvider.Builder.disposeInactivePoolsInBackground(interval, inactivity). SCG never sets this value, so it defaults to Duration.ZERO and the cleanup never runs.
Observed Impact
In our Kubernetes-based Spring Cloud Gateway deployment, we observed:
- 541,106 instances of
reactor.netty.internal.shaded.reactor.pool.SimpleDequePool
- Occupying ~14.3 GB (94.34% of heap)
- GC Root path:
HttpResources → DefaultPooledConnectionProvider → channelPools (ConcurrentHashMap) → SimpleDequePool
- Each
PoolKey.fqdn corresponds to a distinct downstream Pod IP that no longer exists
Heap dump analysis confirmed channelPools.size() equals the number of accumulated SimpleDequePool instances, directly corresponding to the number of distinct downstream IPs seen over the application's lifetime.
Current Workaround
Users must bypass HttpClientProperties entirely and register a custom ConnectionProvider bean:
@Bean
@Primary
public ConnectionProvider gatewayConnectionProvider() {
return ConnectionProvider.builder("gateway")
.maxConnections(500)
.maxIdleTime(Duration.ofSeconds(4))
.maxLifeTime(Duration.ofSeconds(60))
.evictInBackground(Duration.ofSeconds(10))
// Not available via application.properties!
.disposeInactivePoolsInBackground(
Duration.ofMinutes(5), // scan interval
Duration.ofMinutes(5) // inactivity threshold
)
.build();
}
This workaround works but requires users to know the internal reactor-netty API and loses the convenience of application.properties configuration.
Feature Request
Please expose disposeInactivePoolsInBackground in HttpClientProperties (or GatewayProperties) to allow configuration via:
spring.cloud.gateway.httpclient.pool.inactive-pool-dispose-interval=5m
spring.cloud.gateway.httpclient.pool.pool-inactivity=5m
And wire it into HttpClientAutoConfiguration when building the ConnectionProvider:
// In HttpClientAutoConfiguration or similar
if (pool.getInactivePoolDisposeInterval() != null) {
builder.disposeInactivePoolsInBackground(
pool.getInactivePoolDisposeInterval(),
pool.getPoolInactivity()
);
}
Environment
- Spring Cloud Gateway:
3.x (based on Spring Boot 3 / SCG 4.x series)
- Reactor Netty:
1.1.5
- Deployment: Kubernetes with frequent Pod rolling updates
References
Problem Description
Spring Cloud Gateway (SCG) exposes
HttpClientPropertiesto configure the underlying Reactor Netty connection pool, but it does not expose thedisposeInactivePoolsInBackground(interval, inactivity)parameter fromConnectionProvider.Builder.This causes a critical memory leak in environments with dynamic downstream instances (e.g., Kubernetes), where Pod IPs change frequently due to rolling deployments or auto-scaling.
Root Cause Analysis
Reactor Netty's
PooledConnectionProvidermaintains aConcurrentHashMap<PoolKey, InstrumentedPool>namedchannelPools. APoolKeyis composed of:remoteAddress(downstream IP:Port)pipelineKey(channelHash, derived fromHttpClientConfig)Every time a downstream Pod is restarted or replaced, a new IP is assigned → a new
PoolKeyis created → a newSimpleDequePoolis added tochannelPools.The critical problem:
channelPoolsis never cleaned up automatically. The existing configuration options only clean up connections within a pool, not the PoolKey entries (pools themselves):httpclient.pool.max-idle-timehttpclient.pool.max-life-timehttpclient.pool.eviction-intervaldisposeInactivePoolsInBackgroundThe
disposeInactivePoolsInBackgroundmethod exists inPooledConnectionProvider(reactor-netty v1.1.x):This is triggered only when
inactivePoolDisposeIntervalis non-zero, which is set viaConnectionProvider.Builder.disposeInactivePoolsInBackground(interval, inactivity). SCG never sets this value, so it defaults toDuration.ZEROand the cleanup never runs.Observed Impact
In our Kubernetes-based Spring Cloud Gateway deployment, we observed:
reactor.netty.internal.shaded.reactor.pool.SimpleDequePoolHttpResources → DefaultPooledConnectionProvider → channelPools (ConcurrentHashMap) → SimpleDequePoolPoolKey.fqdncorresponds to a distinct downstream Pod IP that no longer existsHeap dump analysis confirmed
channelPools.size()equals the number of accumulatedSimpleDequePoolinstances, directly corresponding to the number of distinct downstream IPs seen over the application's lifetime.Current Workaround
Users must bypass
HttpClientPropertiesentirely and register a customConnectionProviderbean:This workaround works but requires users to know the internal reactor-netty API and loses the convenience of
application.propertiesconfiguration.Feature Request
Please expose
disposeInactivePoolsInBackgroundinHttpClientProperties(orGatewayProperties) to allow configuration via:And wire it into
HttpClientAutoConfigurationwhen building theConnectionProvider:Environment
3.x(based on Spring Boot 3 / SCG 4.x series)1.1.5References
PooledConnectionProvidersource (v1.1.5)