Skip to content

Commit 609a12f

Browse files
committed
feat: virtual thread support
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
1 parent 110199c commit 609a12f

12 files changed

Lines changed: 672 additions & 4 deletions

File tree

docs/content/en/docs/documentation/operations/configuration.md

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,34 @@ Operator operator = new Operator( override -> override
2323
.withLeaderElectionConfiguration(new LeaderElectionConfiguration("bar", "barNS")));
2424
```
2525

26+
### Virtual Threads
27+
28+
Reconciliation is mostly about blocking: talking to the Kubernetes API server or to external
29+
systems. Virtual threads make such blocking calls much cheaper than platform threads, and the
30+
framework can be switched over to them with a single flag:
31+
32+
```java
33+
Operator operator = new Operator(override -> override.withUseVirtualThreads(true));
34+
```
35+
36+
When enabled, reconciliations, dependent resource workflows and the framework's internal
37+
housekeeping (starting the informers, for example) all run on virtual threads.
38+
39+
Enabling virtual threads does **not** remove the concurrency limits, parallelism is configured
40+
exactly as before: `withConcurrentReconciliationThreads(int)` still caps how many reconciliations
41+
run at the same time and `withConcurrentWorkflowExecutorThreads(int)` how many dependent resources
42+
of a workflow are processed concurrently. Only the threads backing those limits change. Since
43+
virtual threads are cheap, these limits can usually be raised significantly compared to what is
44+
reasonable with platform threads.
45+
46+
Two things to keep in mind:
47+
48+
- Virtual threads require Java 21 or later at runtime. When the flag is set on an older JVM, a
49+
warning is logged and platform threads are used instead, so the same configuration works on any
50+
supported Java version.
51+
- A custom `ExecutorService` provided through `withExecutorService(...)` or
52+
`withWorkflowExecutorService(...)` is always used as is, the flag has no effect on it.
53+
2654
## Reconciler-Level Configuration
2755

2856
While reconcilers are typically configured using the `@ControllerConfiguration` annotation, you can also override configuration at runtime when registering the reconciler with the operator. You can either:
@@ -265,6 +293,7 @@ All operator-level keys are prefixed with `josdk.`.
265293
|---|---|---|
266294
| `josdk.check-crd` | `Boolean` | Validate CRDs against local model on startup |
267295
| `josdk.close-client-on-stop` | `Boolean` | Close the Kubernetes client when the operator stops |
296+
| `josdk.use-virtual-threads` | `Boolean` | Run the framework's concurrent work on virtual threads (requires Java 21+ at runtime) |
268297
| `josdk.use-ssa-to-patch-primary-resource` | `Boolean` | Use Server-Side Apply to patch the primary resource |
269298
| `josdk.clone-secondary-resources-when-getting-from-cache` | `Boolean` | Clone secondary resources on cache reads |
270299

operator-framework-core/src/main/java/io/javaoperatorsdk/operator/api/config/ConfigurationService.java

Lines changed: 32 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,6 @@
2020
import java.util.Optional;
2121
import java.util.Set;
2222
import java.util.concurrent.ExecutorService;
23-
import java.util.concurrent.Executors;
2423
import java.util.function.Consumer;
2524

2625
import org.slf4j.Logger;
@@ -228,6 +227,34 @@ default Metrics getMetrics() {
228227
return Metrics.NOOP;
229228
}
230229

230+
/**
231+
* Whether the framework should run the tasks it executes concurrently &mdash; reconciliations,
232+
* dependent workflows and internal housekeeping such as starting the informers &mdash; on virtual
233+
* threads instead of platform threads.
234+
*
235+
* <p>Virtual threads make blocking operations, which is essentially all a reconciler does while
236+
* talking to the Kubernetes API server or to external systems, much cheaper. Enabling them does
237+
* <em>not</em> lift the configured concurrency limits: {@link #concurrentReconciliationThreads()}
238+
* and {@link #concurrentWorkflowExecutorThreads()} still cap how many reconciliations,
239+
* respectively dependent resources, are processed at the same time, they just aren't backed by a
240+
* pool of platform threads anymore. Since virtual threads are cheap, those limits can be set
241+
* considerably higher than what would be reasonable for platform threads.
242+
*
243+
* <p>Requires Java 21 or later at runtime. When enabled on an older JVM, a warning is logged and
244+
* platform threads are used, so that the same configuration works regardless of the Java version
245+
* the operator runs on.
246+
*
247+
* <p>Note that this only affects the executors created by the framework: a custom {@link
248+
* ExecutorService} provided through {@link #getExecutorService()} or {@link
249+
* #getWorkflowExecutorService()} is used as is.
250+
*
251+
* @return {@code true} to use virtual threads, {@code false} (default) to use platform threads
252+
* @since 5.7.0
253+
*/
254+
default boolean useVirtualThreads() {
255+
return false;
256+
}
257+
231258
/**
232259
* Override to provide a custom {@link ExecutorService} implementation to change how threads
233260
* handle concurrent reconciliations
@@ -236,7 +263,8 @@ default Metrics getMetrics() {
236263
* processing
237264
*/
238265
default ExecutorService getExecutorService() {
239-
return Executors.newFixedThreadPool(concurrentReconciliationThreads());
266+
return ExecutorServiceManager.newBoundedExecutorService(
267+
concurrentReconciliationThreads(), useVirtualThreads());
240268
}
241269

242270
/**
@@ -246,7 +274,8 @@ default ExecutorService getExecutorService() {
246274
* @return the {@link ExecutorService} implementation to use for dependent workflow processing
247275
*/
248276
default ExecutorService getWorkflowExecutorService() {
249-
return Executors.newFixedThreadPool(concurrentWorkflowExecutorThreads());
277+
return ExecutorServiceManager.newBoundedExecutorService(
278+
concurrentWorkflowExecutorThreads(), useVirtualThreads());
250279
}
251280

252281
/**

operator-framework-core/src/main/java/io/javaoperatorsdk/operator/api/config/ConfigurationServiceOverrider.java

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,7 @@ public class ConfigurationServiceOverrider {
5050
private KubernetesClient client;
5151
private ExecutorService executorService;
5252
private ExecutorService workflowExecutorService;
53+
private Boolean useVirtualThreads;
5354
private LeaderElectionConfiguration leaderElectionConfiguration;
5455
private String clusterScopedEventNamespace;
5556
private EventRecorder eventRecorder;
@@ -119,6 +120,19 @@ public ConfigurationServiceOverrider withWorkflowExecutorService(
119120
return this;
120121
}
121122

123+
/**
124+
* Makes the framework run the tasks it executes concurrently on virtual threads instead of
125+
* platform threads. Requires Java 21 or later at runtime, see {@link
126+
* ConfigurationService#useVirtualThreads()} for the details.
127+
*
128+
* @param useVirtualThreads {@code true} to use virtual threads
129+
* @return this {@link ConfigurationServiceOverrider} for chained customization
130+
*/
131+
public ConfigurationServiceOverrider withUseVirtualThreads(boolean useVirtualThreads) {
132+
this.useVirtualThreads = useVirtualThreads;
133+
return this;
134+
}
135+
122136
/**
123137
* Replaces the default {@link KubernetesClient} instance by the specified one. This is the
124138
* preferred mechanism to configure which client will be used to access the cluster.
@@ -322,6 +336,11 @@ public boolean closeClientOnStop() {
322336
return overriddenValueOrDefault(closeClientOnStop, ConfigurationService::closeClientOnStop);
323337
}
324338

339+
@Override
340+
public boolean useVirtualThreads() {
341+
return overriddenValueOrDefault(useVirtualThreads, ConfigurationService::useVirtualThreads);
342+
}
343+
325344
@Override
326345
public ExecutorService getExecutorService() {
327346
if (executorService != null) {

operator-framework-core/src/main/java/io/javaoperatorsdk/operator/api/config/ExecutorServiceManager.java

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,37 @@ public class ExecutorServiceManager {
4949
start(configurationService);
5050
}
5151

52+
/**
53+
* Creates the executor service used to run a bounded number of tasks concurrently, either backed
54+
* by virtual threads or by a fixed size pool of platform threads. The concurrency limit is
55+
* enforced in both cases.
56+
*
57+
* @param maxConcurrency the maximal number of tasks executed at the same time
58+
* @param useVirtualThreads whether virtual threads should be used, see {@link
59+
* ConfigurationService#useVirtualThreads()}
60+
* @return the created {@link ExecutorService}
61+
*/
62+
public static ExecutorService newBoundedExecutorService(
63+
int maxConcurrency, boolean useVirtualThreads) {
64+
return VirtualThreads.shouldUse(useVirtualThreads)
65+
? VirtualThreads.newBoundedVirtualThreadExecutor(maxConcurrency)
66+
: Executors.newFixedThreadPool(maxConcurrency);
67+
}
68+
69+
/**
70+
* Creates the executor service used to run an unbounded number of tasks concurrently, either
71+
* backed by virtual threads or by a cached pool of platform threads.
72+
*
73+
* @param useVirtualThreads whether virtual threads should be used, see {@link
74+
* ConfigurationService#useVirtualThreads()}
75+
* @return the created {@link ExecutorService}
76+
*/
77+
public static ExecutorService newUnboundedExecutorService(boolean useVirtualThreads) {
78+
return VirtualThreads.shouldUse(useVirtualThreads)
79+
? VirtualThreads.newVirtualThreadPerTaskExecutor()
80+
: Executors.newCachedThreadPool();
81+
}
82+
5283
/**
5384
* Uses cachingExecutorService from this manager. Use this only for tasks, that don't have dynamic
5485
* nature, in sense that won't grow with the number of inputs (thus kubernetes resources)
@@ -135,7 +166,8 @@ public ScheduledExecutorService scheduledExecutorService() {
135166
public synchronized void start(ConfigurationService configurationService) {
136167
if (!started) {
137168
this.configurationService = configurationService; // used to lazy init workflow executor
138-
this.cachingExecutorService = Executors.newCachedThreadPool();
169+
this.cachingExecutorService =
170+
newUnboundedExecutorService(configurationService.useVirtualThreads());
139171
this.scheduledExecutorService = Executors.newScheduledThreadPool(0);
140172
this.executor = new InstrumentedExecutorService(configurationService.getExecutorService());
141173
started = true;
Lines changed: 178 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,178 @@
1+
/*
2+
* Copyright Java Operator SDK Authors
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
package io.javaoperatorsdk.operator.api.config;
17+
18+
import java.lang.invoke.MethodHandle;
19+
import java.lang.invoke.MethodHandles;
20+
import java.lang.invoke.MethodType;
21+
import java.util.List;
22+
import java.util.concurrent.AbstractExecutorService;
23+
import java.util.concurrent.ExecutorService;
24+
import java.util.concurrent.Executors;
25+
import java.util.concurrent.Future;
26+
import java.util.concurrent.Semaphore;
27+
import java.util.concurrent.TimeUnit;
28+
import java.util.concurrent.atomic.AtomicBoolean;
29+
30+
import org.slf4j.Logger;
31+
import org.slf4j.LoggerFactory;
32+
33+
import io.javaoperatorsdk.operator.OperatorException;
34+
35+
/**
36+
* Creates the virtual thread based executors used when {@link
37+
* ConfigurationService#useVirtualThreads()} is enabled.
38+
*
39+
* <p>The SDK is compiled for Java 17, in which virtual threads don't exist yet, so {@code
40+
* Executors.newVirtualThreadPerTaskExecutor()} is looked up reflectively and is only available when
41+
* the operator actually runs on Java 21 or later.
42+
*/
43+
final class VirtualThreads {
44+
45+
private static final Logger log = LoggerFactory.getLogger(VirtualThreads.class);
46+
47+
private static final MethodHandle NEW_VIRTUAL_THREAD_PER_TASK_EXECUTOR = lookupFactoryMethod();
48+
private static final AtomicBoolean UNSUPPORTED_WARNING_LOGGED = new AtomicBoolean();
49+
50+
private VirtualThreads() {}
51+
52+
private static MethodHandle lookupFactoryMethod() {
53+
try {
54+
return MethodHandles.publicLookup()
55+
.findStatic(
56+
Executors.class,
57+
"newVirtualThreadPerTaskExecutor",
58+
MethodType.methodType(ExecutorService.class));
59+
} catch (NoSuchMethodException | IllegalAccessException e) {
60+
log.debug("Virtual threads are not available on this JVM", e);
61+
return null;
62+
}
63+
}
64+
65+
/** Whether the JVM the operator runs on supports virtual threads, i.e. is Java 21 or later. */
66+
static boolean isSupported() {
67+
return NEW_VIRTUAL_THREAD_PER_TASK_EXECUTOR != null;
68+
}
69+
70+
/**
71+
* Whether virtual threads should effectively be used, i.e. they were requested through {@link
72+
* ConfigurationService#useVirtualThreads()} <em>and</em> the JVM supports them. Requesting them
73+
* on a JVM that doesn't support them is only warned about, so that the same configuration can be
74+
* used regardless of the Java version the operator ends up running on, the only consequence being
75+
* that platform threads are used instead. Concurrency limits are enforced either way.
76+
*/
77+
static boolean shouldUse(boolean requested) {
78+
if (!requested || isSupported()) {
79+
return requested;
80+
}
81+
if (UNSUPPORTED_WARNING_LOGGED.compareAndSet(false, true)) {
82+
log.warn(
83+
"Virtual threads were requested but are not supported by the JVM in use (Java {}, Java 21"
84+
+ " or later is required). Falling back to platform threads.",
85+
Runtime.version().feature());
86+
}
87+
return false;
88+
}
89+
90+
/** An unbounded executor starting a new virtual thread for each submitted task. */
91+
static ExecutorService newVirtualThreadPerTaskExecutor() {
92+
if (!isSupported()) {
93+
throw new OperatorException(
94+
"Virtual threads are not supported by the JVM in use, Java 21 or later is required");
95+
}
96+
try {
97+
return (ExecutorService) NEW_VIRTUAL_THREAD_PER_TASK_EXECUTOR.invokeExact();
98+
} catch (Throwable e) {
99+
throw new OperatorException("Couldn't create a virtual thread per task executor", e);
100+
}
101+
}
102+
103+
/**
104+
* A virtual thread based executor executing at most {@code maxConcurrency} tasks at the same
105+
* time, the equivalent of a fixed size platform thread pool.
106+
*/
107+
static ExecutorService newBoundedVirtualThreadExecutor(int maxConcurrency) {
108+
return new BoundedExecutorService(newVirtualThreadPerTaskExecutor(), maxConcurrency);
109+
}
110+
111+
/**
112+
* Limits how many of the tasks submitted to the wrapped executor run at the same time.
113+
*
114+
* <p>A thread is started for each task as soon as it is submitted, the task then waits for a
115+
* permit before it actually runs. This only makes sense with virtual threads, which are cheap
116+
* enough to be parked in large numbers, and has the property that submitting a task never blocks
117+
* the submitting thread, just like queuing it on a fixed size platform thread pool wouldn't.
118+
*/
119+
private static final class BoundedExecutorService extends AbstractExecutorService {
120+
121+
private final ExecutorService delegate;
122+
private final Semaphore permits;
123+
124+
private BoundedExecutorService(ExecutorService delegate, int maxConcurrency) {
125+
this.delegate = delegate;
126+
// fair, so that tasks run roughly in submission order as they would on a thread pool
127+
this.permits = new Semaphore(maxConcurrency, true);
128+
}
129+
130+
@Override
131+
public void execute(Runnable command) {
132+
delegate.execute(
133+
() -> {
134+
try {
135+
permits.acquire();
136+
} catch (InterruptedException e) {
137+
Thread.currentThread().interrupt();
138+
// shutdownNow interrupted us before the task even started: cancel it so that whoever
139+
// waits on the associated future isn't left hanging
140+
if (command instanceof Future) {
141+
((Future<?>) command).cancel(false);
142+
}
143+
return;
144+
}
145+
try {
146+
command.run();
147+
} finally {
148+
permits.release();
149+
}
150+
});
151+
}
152+
153+
@Override
154+
public void shutdown() {
155+
delegate.shutdown();
156+
}
157+
158+
@Override
159+
public List<Runnable> shutdownNow() {
160+
return delegate.shutdownNow();
161+
}
162+
163+
@Override
164+
public boolean isShutdown() {
165+
return delegate.isShutdown();
166+
}
167+
168+
@Override
169+
public boolean isTerminated() {
170+
return delegate.isTerminated();
171+
}
172+
173+
@Override
174+
public boolean awaitTermination(long timeout, TimeUnit unit) throws InterruptedException {
175+
return delegate.awaitTermination(timeout, unit);
176+
}
177+
}
178+
}

operator-framework-core/src/test/java/io/javaoperatorsdk/operator/api/config/ConfigurationServiceOverriderTest.java

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -152,6 +152,13 @@ void threadCountConfiguredProperly() {
152152
.isEqualTo(14);
153153
}
154154

155+
@Test
156+
void virtualThreadsAreDisabledByDefaultAndCanBeOverridden() {
157+
assertThat(config.useVirtualThreads()).isFalse();
158+
assertThat(new ConfigurationServiceOverrider(config).withUseVirtualThreads(true).build())
159+
.returns(true, ConfigurationService::useVirtualThreads);
160+
}
161+
155162
@SuppressWarnings("rawtypes")
156163
@Test
157164
void dependentResourceFactoryDefaultsToTheSharedOneAndCanBeOverridden() {

0 commit comments

Comments
 (0)