Skip to content

fix: prevent Netty command center from blocking JVM shutdown - #3642

Open
lymerin wants to merge 2 commits into
alibaba:1.8from
lymerin:fix-2964-netty-shutdown
Open

fix: prevent Netty command center from blocking JVM shutdown#3642
lymerin wants to merge 2 commits into
alibaba:1.8from
lymerin:fix-2964-netty-shutdown

Conversation

@lymerin

@lymerin lymerin commented Aug 11, 2026

Copy link
Copy Markdown

Describe what this PR does / why we need it

The Netty HTTP command center creates boss and worker event-loop threads with Netty's default non-daemon thread factory. Since the global command center is not always tied to an application lifecycle callback, these threads can keep the JVM alive after a Spring ApplicationContext is closed.

In addition, explicit shutdown previously closed the server channel asynchronously and immediately interrupted the executor via shutdownNow(). If the interruption occurred before the channel close completed, closeFuture().sync() threw InterruptedException, and a normal shutdown was incorrectly reported as a server startup failure. A stop request racing with a successful bind could also encounter a null channel or be lost.

This PR prevents the Netty command center from blocking JVM shutdown and avoids exception noise during normal explicit shutdown.

Does this pull request fix one issue?

Fixes #2964

Describe how you did it

  • Use Netty DefaultThreadFactory with daemon threads for the boss and worker event-loop groups.
  • Replace shutdownNow() with shutdown() so normal channel closure is not interrupted.
  • Safely publish the server channel and stop request with volatile state.
  • Return safely when stop is requested before the server task starts.
  • Close a newly bound channel immediately when a stop request raced with a successful bind.
  • Remove the duplicate direct printStackTrace() call while preserving RecordLog reporting for real startup failures.
  • Add regression tests for JVM exit, daemon thread attributes, explicit shutdown, and stop-before-start behavior.

Describe how to verify it

Run the Netty HTTP transport module tests with JDK 17:

mvn --batch-mode -pl sentinel-transport/sentinel-transport-netty-http -am test

The regression tests verify that:

  • the command center executor and Netty boss/worker threads are daemon threads;
  • a child JVM exits normally without an explicit stop() call after the server starts;
  • explicit stop() does not produce InterruptedException or a false startup failure;
  • stopping before the asynchronous server task starts is safe and does not leak the listening port;
  • the command center still handles real HTTP command requests.

All 14 tests in the module pass.

Special notes for reviews

This change is limited to sentinel-transport-netty-http and does not add a Spring dependency or lifecycle hook.

The existing port-binding retry policy is unchanged. stop() remains non-blocking and is not intended to be a synchronous resource-termination barrier.

@CLAassistant

CLAassistant commented Aug 11, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@oss-sentinel-ai oss-sentinel-ai left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Fixes the Netty command center hanging JVM shutdown by switching to daemon event-loop threads and dropping the spurious InterruptedException on explicit shutdown. The direction is right, but the new stop/bind race guard is not correctly synchronized and the server becomes permanently one-shot — requesting changes before merge.


Automated review by github-manager-bot


public void close() {
channel.close();
stopped = true;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The stop/bind race guard uses two independent volatile fields (stopped and channel), which does not guarantee cross-visibility: start() may still observe stopped == false after assigning the channel and leave the server listening. Synchronize the stop flag and channel assignment under a common monitor, or hold both in a single atomic state object, so a close() request can never be lost.

ChannelFuture channelFuture = null;
// loop for an successful binding
// Retry binding on incremented ports until a port is available.
while (true) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bind retry loop does not recheck stopped between failed binds. If close() is called while retrying, the loop continues scanning ports until a bind succeeds (or indefinitely), delaying shutdown. Check stopped in the catch/retry path and break out promptly.

@@ -102,7 +124,11 @@ private int getNewPort(int basePort, int retryCount) {
}

public void close() {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting stopped = true without resetting it makes HttpServer one-shot: after CommandCenter.stop(), a later start() will silently do nothing. This changes the previous lifecycle semantics; please document the intended one-shot behavior or reset stopped when a restart is appropriate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

When spring-boot application context closed sentinel still has some not closed non-daemons thread

3 participants