[KYUUBI #7655][SERVER] Support virtual threads in the binary frontend

[wangzhigang1999]

Why are the changes needed?

Virtual threads are mature in JDK 21, which Kyuubi already supports. As an I/O-bound gateway, Kyuubi is a good fit for this execution model.

This PR makes a small first step by adding optional virtual-thread support to the server-side Thrift Binary frontend. It is disabled by default and does not affect engine frontends or other executors.

The implementation preserves the worker concurrency limit and rejection behavior while retaining compatibility with older JDKs.

See #7655 for the motivation and benchmark results.

How was this patch tested?

• Targeted suites passed on JDK 11 and JDK 21.
• dev/reformat
• git diff --check
• Kyuubi Server fast package build.
• Five-minute A/B test with real Spark SQL.

Was this patch assisted by generative AI tooling?

Assisted-by: OpenAI Codex (GPT-5)

[pan3793]

I remember we have lots of synchronized in the codebase, and will it cause issues on JDK 21?

[wangzhigang1999]

Yes, I also tried a more aggressive version that converted the entire SQL pipeline—including the Spark SQL engine—to virtual threads.
At that time, I found that synchronous Log4j output caused pinning, which even led to hangs under the default carrier thread count. Therefore, I significantly narrowed the scope of the first PR.
Your reminder is crucial. If you also agree that we can introduce virtual threads, I will conduct benchmarks on larger-scale real-world datasets, which would make the case more compelling.If necessary, some synchronized blocks can also be optimized.

[pan3793]

maybe we can benchmark with JDK 25 #7657

[wangzhigang1999]

I’ll give it a try.