Looming Marvelous - Virtual Threads in Java Javaland.pdf

Project Loom - Einfachere Nebenläuﬁgkeit in Java
Ein Thread, ist ein Thread, ist ein (virtueller) Thread.
Code Examples

Leading developer relations
& community at the
totally best
graph database
Who's talking?
Michael Hunger
Learning new things every
day
Having fun with code,
open-source, people
Java
Working with Java since
1995
Jug-Saxony
JavaChampion
Neo4j TechnoGirls
Supporting girls
to learn to code

Welcome
Java 20
Mar 21, 2023
sdk install java 20-open
openjdk 20 2023-03-21 build 20+36-2344

Why are we talking
about this ?
Concurrency ?
Parallelism?

There is a lot to do in a Java Program
Image Credit Marcus Biel / cleancodeacademy

● Process and answer requests
● Compute
● I/O: Read and write ﬁles
● Read from databases and network
● Synchronization
● Render
● Logging / Monitoring / Metrics
● Garbage collection
There is a lot to do! And limited resources
● Developer Brain
● CPU / Hyperthreading
● Memory
● I / O
● Network
● Disk

Been There - Seen That
Java 1.1 - 1997
Green Threads
Java 1.3 - 1998
Native Threads
Java 1.5 - 2004
Thread Pools / Executors
Java 8 2014-2016
CompletableFuture
Rx-Java
Reactive Streams
Reactor
2017-2018
Java 9
Flow
Kotlin Coroutines - Stable
Java 19, 2022
JEP 425 Virtual Threads
Preview
Java 1.7 2011
ForkJoinPool
Parallel Streams
Java 20, 2023
JEP 436 Virtual Threads
2nd Preview

A Thread
● Asynchronous Unit of Execution
● Multiple Threads running inside a Process
● One Thread at a time on a CPU
● Inside a thread - synchronous execution
● Switching - Switch/Save Registers / Invalidate Caches …
● Cheaper than process switching, HW support
● green (<1.3), platform/native, now virtual Threads

Green Threads vs. Native Threads
Green Threads
● User Level Threads
● Simulated Multithreading
● Runs on single (LW)Process / CPU
● Scheduled by VM, not OS
● Lots of HW context switching
● "slow"
● Abandoned in Java 1.3
● Management / State overhead
Native Threads
● Mapped to all HW Threads (HT)
● Limited to HW concurrency
● blocking, synchronous execution
● Since Java 1.3 (Solaris - 1.2)
● Better on I/O & context-switching
● Faster than process based concurrency
And now we have virtual Threads…

Java Concurrency looks
easier than it is
Devil is in the details
(Brian Goetz - JCP book)

Hello Threading World
Example Code & Run
1. new Thread().start()
2. ThreadPool/Executor
3. Reactive Streams/Reactor/Akka
4. Kotlin Coroutines
5. Loom - Virtual Threads

Java Threads & Thread Pools
var t = new Thread(() -> System.out.println("Hello World"));
t.start();
t.join();
// new! Thread.Builder
var t = Thread.ofPlatform().start(
() -> System.out.println("Hello new World"));
t.join();

Thread Pools
try (var executor = Executors.newFixedThreadPool(5)) {
IntStream.range(0, 50).forEach(i -> {
executor.submit(() ->
System.out.println("Hello Platform Thread "+i+" "+Thread.currentThread())
);
});
} // executor.close()

Thread Pools
// takes a long time
try (var executor = Executors.newFixedThreadPool(10)) {
IntStream.range(0, 10_000).forEach(i -> {
executor.submit(() -> {
Thread.sleep(Duration.ofSeconds(1));
return i;
});
});
} // executor.close()

// parallel stream
IntStream.range(1,10).parallel()
.mapToObj( i -> "Hello World "+i)
.forEach(System.out::println);
Parallel Streams

// Completable Future
var cf = CompletableFuture.completedFuture("complex")
.thenApplyAsync(String::toUpperCase)
.thenCombine(
CompletableFuture.completedFuture("CODE")
.thenApplyAsync(String::toLowerCase),
(s1, s2) -> s1 + s2);
cf.join()
Completable Future

Reactive Programming
Reactive Java, Reactor, RxJava, Akka
String key = "message";
Mono<String> r = Mono.just("Hello")
.flatMap(s -> Mono.deferContextual(ctx ->
Mono.just(s + " " + ctx.get(key))))
.contextWrite(ctx -> ctx.put(key, "World"));
StepVerifier.create(r).expectNext("Hello World").verifyComplete();
Reactor Documentation

Kotlin Coroutines
Documentation
fun main() = runBlocking { // this: CoroutineScope
launch { doWorld() }
println("Hello")
}
// suspending function
suspend fun doWorld() {
delay(1000L)
println("World!")
}

Blocking vs. Non-Blocking vs. Continuations
Blocking
● linear program-flow
● execute what you
wrote, easy to reason
● blocks on intensive
operations
● inefficient use of
resources
(utilized/blocked)
● classical Java Threading
Continuation
● ability to capture
computation so far and
continue later
● explicit continuations
(await, async, yield)
● implicit continuations (on
entry-points to blocking
ops)
● simpler API
● hard work is in the
implementation
● Kotlin, JS, Loom
Non-Blocking
● DSL for describing a
processing
● underlying reactive engine
● data flow
● resource efficient
● hard to reason, debug
unit-test, profile, maintain
● large complex API
● difficult to correlate
operations
● RxJava, Reactive Streams,
Akka

Concurrency & Parallelism help us make
more eﬃcient use of existing resources

Concurrency & Parallelism (in Java)

Concurrency vs. Parallelism
Concurrency
● Ability to execute many (diﬀerent) tasks
and make progress on all of them
● Seemingly simultaneously (e.g. on 1 CPU)
Parallel Concurrent Execution
● Multiple tasks are executed concurrently
(at the same time) AND
● Multiple CPUs are used to execute tasks
in parallel
● Most common today
Parallel Execution
● Utilize more than 1 CPU/Thread to
progress multiple tasks simultaneously
Parallelism
● Ability to divide and conquer a single
task into subtasks that can be executed
in parallel
source: jenkov.com

Concurrent
Parallel Execution
Parallel Concurrent Execution
CPU
Task 1
Task 2
CPU Task 1
Task 2
CPU
CPU
Task 1
Task 2
CPU
Task 3
Task 4
Parallelism
CPU
Thread 1
Thread 2
CPU
Thread 3
Thread 4

Parallel Execution Challenges
● Context Switches / Caches / Branch Predictions
● Race Conditions
● Mutable data / visibility
● Deadlocks
● Starvation
● Resource over- / underutilization
● Immutability / Ownership
● Reasoning / Debugging / Logging
● State-Management
● Execution Depedencies

Project Loom
JEP 425 / JEP 436
Image OpenJDK / Sharat Chander

JEP Process
JEP 425
1st Preview
Java 19
Sept 2022
Delivered
JEP 436
2nd Preview
Java 20
Mar 2023
Delivered
JEP Draft
Finalization
Java 21
Sept 2023
Submitted
● Thread Locals Support
● Virtual Thread API
● Thread.Builder
● VirtualThreadPoolExecutor
● Continuations impl
● StructuredConcurrency (inc.)
● StructuredConcurrency
● Scoped Variables

JEP 425/436 - Project Loom - openjdk.org/jeps/425 …/436

Goals
● Enable server applications written in the
simple thread-per-request style to
scale with near-optimal hardware
utilization.
● Enable existing code that uses the
java.lang.Thread API to adopt
virtual threads with minimal
change.
● Enable easy troubleshooting,
debugging, and profiling of virtual
threads with existing JDK tools.
Loom - Goals
Non-Goals
● It is not a goal to remove the traditional
implementation of threads, or to
silently migrate existing
applications to use virtual threads.
● It is not a goal to change the basic
concurrency model of Java.
● It is not a goal to offer a new data
parallelism construct in either the
Java language or the Java libraries. The
Stream API remains the preferred way
to process large data sets in parallel.

● Continuations internally in the JVM
● Reimplementation of Networking / IO Code in JVM
● lightweight Virtual Thread, same API as java.util.Thread
● Thread.Builder
● VirtualThreadExecutors
● Auto-Closeable Excecutors
● Structured Concurrency (StructuredTaskScope)
Java 20
● Scoped Values + their support in StructuredTaskScope
● StructuredTaskScope still incubator
What's in the boxcup?

Virtual Thread
● same, stable API as traditional thread (deprecations will be removed)
● handled diﬀerently during blocking operations
● lightweight (300 bytes) like a Runnable, JVM can execute millions
● temporarily bound to a platform (carrier) thread
● on each blocking/parking operation -> Continuation yielding
● stack is copied to heap
● on resume, stack copied back and
● execution resumed on diﬀerent carrier Thread
● uses a separate ForkJoinPool (FJP), to also prevent starving
-Djdk.defaultScheduler.parallelism=N

var threads =
IntStream.range(0,10).mapToObj(i ->
Thread.ofVirtual().start(() -> { // or Thread.startVirtualThread(runnable)
System.out.println("Hello Virtual Thread "+i+" "+Thread.currentThread());
})).toList();
for (Thread t : threads) {
t.join();
}
Java Virtual Threads - Builder
Hello Virtual Thread 3 VirtualThread[#10079]/runnable@ForkJoinPool-1-worker-18

Java Virtual Threads - Executor
try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
IntStream.range(0, 100_000).forEach(i -> {
executor.submit(() -> {
Thread.sleep(java.time.Duration.ofSeconds(1));
return i; // callable, throws Exception
});
});
} // executor.close() is called implicitly, and waits

New Rule:
If in doubt
Start a new virtual Thread

// java --enable-preview --source 20 LoomServer.java
// echo -n 'Hello Loom' | nc -n 127.0.0.1 2000
import java.io.*;
import java.net.*;
import java.util.concurrent.*;
public class LoomServer {
public static void main(String...args) throws IOException {
try (var ss = new ServerSocket(2000);
var pool = Executors.newVirtualThreadPerTaskExecutor()) {
while (true) {
var socket = ss.accept();
pool.execute(() -> {
try (var s = socket;
var in = s.getInputStream();
var out = s.getOutputStream()) {
byte b = -1;
while ((b = (byte)in.read()) != -1) {
out.write(b+1);
}
} catch(IOException ioe) {}
});

Under The Hood
● Virtual Threads run on (diﬀerent) Platform Threads
● They use a separate Fork Join Pool
● Instead of blocking (IO, networking, sleep, synchronization) they are
yielding control
● Blocking Code in the JVM refactored to use Continuations
● Continuations move stack from Platform Thread to Heap
● Can pick up later on another thread
● Except when using addresses or native code

Example Thread.sleep()
● Thread.sleep()
○ VThread.sleepNanos()
■ VThread.doSleepNanos()
● VTread.parkNanos()
○ VThread.yieldContinuation()
■ unmount()
■ Continuation.yield()
static final ContinuationScope VTHREAD_SCOPE =
new ContinuationScope("VirtualThreads");
@ChangesCurrentThread
private boolean yieldContinuation() {
boolean notifyJvmti = notifyJvmtiEvents;
// unmount
if (notifyJvmti) notifyJvmtiUnmountBegin(false);
unmount();
try {
return Continuation.yield(VTHREAD_SCOPE);
} finally {
// re-mount
mount();
if (notifyJvmti) notifyJvmtiMountEnd(false);
}
}
private void unmount() {
// set Thread.currentThread() to return the platform thread
Thread carrier = this.carrierThread;
carrier.setCurrentThread(carrier);
// break connection to carrier thread, synchronized with interrupt
synchronized (interruptLock) {
setCarrierThread(null);
}
carrier.clearInterrupt();
}

Caveats or when does it not work?
● when stack cannot be moved to heap
● if it contains memory addresses
(synchronized) -> use ReentrantLock!
● calls c-code
● File I/O
● DNS (Windows)
● then the task stays pinned to a platform
thread
● to avoic exhaustion the FJP creates new
temporary platform threads
● can track with
-Djdk.tracePinnedThreads=full
example by A. Sundararajan

● Large pull request (#8166) touching 1333 ﬁles
● Thread.Builder, virtualThreadPerTaskExecutor
● refactor all blocking/parking code to use Continuations for Virtual Threads
○ Network I/O,
○ Locks
○ Thread.sleep
● replace c-code where possible with Java code (e.g. in Method ->
MethodHandles)
● not (yet):
○ File I/O (waiting for io_uring)
○ synchronized, due to address usage
JDK Changes

Continuations on the long way to Loom

"A continuation is a callback function k that represents the
current state of the program's execution.
More precisely, the continuation k is a function of one
argument, namely the value that has been computed so far,
that returns the ﬁnal value of the computation after the rest
of the program has run to completion."
A continuation

program
program
Stack Heap
program
continued
Stack Heap
Carrier
Threads
Continuation
yield

import jdk.internal.vm.Continuation;
import jdk.internal.vm.ContinuationScope;
var scope = new ContinuationScope("scope");
var c = new Continuation(scope, () -> {
System.out.println("Started");
Continuation.yield(scope);
System.out.println("Running");
Continuation.yield(scope);
System.out.println("Still running");
});
System.out.println("Start");
int i=0;
while (!c.isDone()) {
c.run();
System.out.println("Running "+i+" result "+c.isDone());
i++;
}
System.out.println("End"); Continuations in the JDK
Start
Started
Running 0 result false
Running
Running 1 result false
Still running
Running 2 result true
End

Structured
Concurrency?!?
JEP 428 / JEP 437

Launching Millions of Threads is no better
than GOTO
Nathaniel Smith

Structured Concurrency (JEP 428)
Goals
■ Improve the maintainability,
reliability, and observability of
multithreaded code.
■ Promote a style of concurrent
programming which can
eliminate common risks arising
from cancellation and shutdown,
such as thread leaks and
cancellation delays.
Non-Goals
■ It is not a goal to replace any of the concurrency
constructs in the java.util.concurrent package,
such as ExecutorService and Future.
■ It is not a goal to define the definitive structured
concurrency API for Java. Other structured
concurrency constructs can be defined by
third-party libraries or in future JDK releases.
■ It is not a goal to define a means of sharing
streams of data among threads (i.e., channels). We
might propose to do so in the future.
■ It is not a goal to replace the existing thread
interruption mechanism with a new thread
cancellation mechanism. We might propose to do so
in the future.

Structured Concurrency (JEP 428)
● with so many threads running you need management and control
structures
● since JDK 5 no direct interaction with Threads but submit to
ExecutorService -> Future
● ExecutorService is still unstructured
● Multi-Thread Control structures are missing in Java (like Erlang/Akka)
○ even if they exist in the business process
○ no task->subtask relationships between threads
○ every thread can read from a future or submit to an executor
● Loom keeps this model -> Structure is missing
● what happens when a (child or parent) thread fails?

Comparison
Response handle()
throws ExecutionException, InterruptedException{
Future<String> user =
ex.submit(() -> findUser());
Future<Integer> order =
ex.submit(() -> fetchOrder());
// Join findUser
String theUser = user.get();
// Join fetchOrder
int theOrder = order.get();
return new Response(theUser, theOrder);
}

StructuredTaskScope Example
try ( var scope = new StructuredTaskScope<String>() ) {
var future1 = scope.fork(task1);
var future2 = scope.fork(task2);
scope.join();
return switch (future1.state()) {
case Future.SUCCESS -> future1.resultNow();
case Future.FAILED -> future1.exceptionNow();
}
} API StructuredTaskScope

StructuredTaskScope.ShutdownOnSuccess
import jdk.incubator.concurrent.*;
try ( var scope = new StructuredTaskScope.ShutdownOnSuccess<String>() ) {
IntStream.range(0,10).forEach(i ->
scope.fork(() -> String.valueOf(i)));
scope.join();
// first returning wins, exception if none did
System.out.println(scope.result());
}
● ShutdownOnFailure - same just for fail-fast API StructuredTaskScope

● more explicit clarity of what happens
● specialized, auto-closeable, short-lived execution-scope
● like an executor, but uses virtual Threads and FJP
● submit tasks to it (fork()) all work on behalf of scope
● StructuredTaskScope<T>()
● Future<T> future = scope.fork(task);
● scope.join() -> returns when all tasks are complete
● switch on future.state() (FAILED, RUNNING, SUCCESS, CANCELLED)
○ future.resultNow() / future.exceptionNow()
● better with specialized implementations (ﬁrst-one-wins or fail-fast)
○ if one fails, others are cancelled via ShutdownOnFailure cancellation policy
● auto-cleanup
Scope - virtual thread launcher

● to implement your own "structured business logic"
● Subclass StructuredTaskScope
● override handleComplete(Future<T>)
● depending on future-state, do what you need to do
● is called concurrently, needs to use thread safe instance state
● custom result method, reduce state to a result (or Exception)
● Concern: Still a lot of multi-threaded technical infrastructure complexity
Your own Scope
JEP Café #13

Scoped Values!
since Java 20
JEP 429
(was ExtendedLocal)

Thread Locals - we all know and love
● since Java 1.2
● Thread local/safe access (guaranteed)
● Store mutable state on a thread
○ Side-channel communication
● Can leak memory, until garbage collected or remove() called
● Inherited by child threads
● Smaller issue with platform threads
● Work with Virtual Threads, but
● Massive issue with huge amount of virtual threads

Scoped Values for the Rescue (JEP 429 - Incubated)

Immutable one-way communication of data to called runnables
● Deﬁne value - types
● Set up scope(s) with initializer per variable with .where(VAR, value)
● Bindings only valid during run(lambda) scope
private static final ScopedValue<..> VAL = ScopedValue.newInstance();
// Set up scope, each where yields a new scope
ScopedValue.where(VAL, <value>).where(VAL2, <value2>)
// use scope
.run(() -> { ... VAL.get() ... call methods ... });
Scoped Values for the Rescue
API Documentation

Example
import jdk.incubator.concurrent.*;
private static final ScopedValue<String> VERSION =
ScopedValue.newInstance();
Runnable runnable = () -> System.out.println(VERSION.get());
ScopedValue.where(VERSION, "Java 19", () ->
System.out.println(VERSION.get()));
var s = ScopedValue.where(VERSION, "Java 20");
s.run(runnable); // Java 20
s.where(VERSION, "Java 21").run(runnable); // Java 21
s.run(runnable); // Java 20

● Stacktrace in IDE-Debuggers (as expected)
● Patches in Java Debug Wire Protocol (JWDP) & Java Debugger Interface (JDI)
● Challenge - Display Millions of Threads
● Structured concurrency can help here too (Tree-Display)
● JFR should work - match allocations, method calls, etc. to virtual Threads
● Gaps in Thread API:
○ list all running threads
○ carrier <-> virtual thread
Loom Debugging - Just regular!?
wiki.openjdk.org/display/loom/Debugger+Support

Loom Demos &
Comparisions
github.com/ebarlas
Web-Backend
Game of Life
5M persistent connections
Elliot Barlas - MicroHttp - LogMeIn

Performance Comparison
Web-Server calling Backend
github.com/ebarlas/project-loom-comparison

WebServer
● Browser (ab)
● Web-Server calling
● 3x Backend (0,3s latency)
○ Authentication
○ Authorization
○ Database
● 3 implementations
○ Platform Threads
○ Asynchronous
○ Virtual Threads

public void handle(Request request, Consumer<Response> callback) {
executorService.execute(() -> callback.accept(doHandle(request)));
}
Response doHandle(Request request) {
var token = request.header("Authorization");
var authentication = sendRequestFor("/authenticate?token=" + token, …);
var authorization = sendRequestFor("/authorize?id=" + authentication.userId(), ..);
var meetings = sendRequestFor("/meetings?id=" + authentication.userId(), …);
var headers = List.of(new Header("Content-Type", "application/json"));
return new Response(200, "OK", headers, Json.toJson(meetings));
}
<T> T sendRequestFor(String endpoint, Class<T> type)
throws IOException, InterruptedException {
URI uri = URI.create("http://%s%s".formatted(backend, endpoint));
var request = HttpRequest.newBuilder().uri(uri).GET().build();
HttpResponse<String> response = httpClient.send(request,
HttpResponse.BodyHandlers.ofString());
if (response.statusCode() != 200) {
throw new RuntimeException("error occurred contacting "+endpoint);
}
return Json.fromJson(response.body(), type);
}
Same old Executor (for both Platform & Virtual Threads)

java --enable-preview -jar nima/target/example-nima-blocking.jar
2022.09.22 02:36:05.204 Logging at initialization configured using classpath: /logging.properties
2022.09.22 02:36:05.394 [0x6e82e640] http://127.0.0.1:8080 bound for socket '@default'
2022.09.22 02:36:05.396 [0x6e82e640] async writes, queue length: 32
2022.09.22 02:36:05.403 Níma server started all channels in 8 milliseconds. 244 milliseconds since JVM startup. Java
19+36-2238
ab -n 10000 -c 8 http://127.0.0.1:8080/one
Concurrency Level: 8
Time taken for tests: 0.848 seconds
Complete requests: 10000
Requests per second: 11797.14 [#/sec] (mean)
Time per request: 0.678 [ms] (mean)
Time per request: 0.085 [ms] (mean, across all concurrent requests)
Transfer rate: 0.00 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 0 0 0.0 0 0
Waiting: 0 0 0.0 0 0
Total: 0 0 0.0 0 1
Nima Test

Helidon Nima
Note: In either case, you should not
“obstruct” the thread. Obstruction is a
long-term, full utilization of the thread.
In a reactive framework this would
consume one of the event loop threads
effectively stopping the server. In
blocking (Níma) this may cause an
issue with the “pinned thread”. In both
cases this can be resolved by
off-loading the heavy load to a
dedicated executor service using
platform threads.

Performance comparable to Netty pipelined
Note: What we can see from these numbers
(and what is our goal with Níma) is that we
can achieve performance comparable to a
minimalist Netty server, while maintaining
a simple, easy to use programming model.

Game of Life
CSP - Communicating Sequential Processes
github.com/ebarlas/game-of-life-csp

CSP - Not possible in Java before
● Channel (BlockingQueue<Boolean>) to exchange information
● Grid - Channel<boolean[][]>
● Each Cell has
○ a virtual thread
○ channels for ticks and results (width x height x 2)
○ one channel per neighbour (~ width x height x 8)
○ aka a LOT of channels/queues
Wikipedia CSP

Cell's biological Clock - "Life"
private void run() {
while (true) {
tickChannel.take(); // wait for tick stimulus
// announce liveness to neighbors
outChannels.forEach(ch -> ch.put(alive));
// receive liveness from neighbors
int neighbors = inChannels.stream()
.map(Channel::take)
.mapToInt(b -> b ? 1 : 0).sum();
// calculate next state based on game of life rules
alive = alive && neighbors == 2 || neighbors == 3;
// announce resulting next state
resultChannel.put(alive);
}
}
● take / put -> block -> suspend
● exchange via channel/queue
● write only to local state

Looming Marvelous - Virtual Threads in Java Javaland.pdf

Mario Fusco 🚀
Improved channel datastructures
yield massive improvements
speakerdeck.com/mariofusco
Speaking Tour

● JEP 425 - Virtual Threads (Java 19)
● JEP 436 - Virtual Threads (Java 20)
● JEP 428 - Structured Concurrency
● JEP Café #12 - 10M Threads
● JEP Café #13 - Loom Tutorial
● Heinz Kabutz Loom Video
● JavaSpektrum Loom (me)
● Loom Lab (Nicolai Parlog)
● Million Virtual Threads
● Virtual Threads PR
● Loom Talk Pressler Bates Devoxx
● Loom Deep Dive: Paumard Forax Devoxx
● Loom Wiki
● Inside Java Loom
● Loom Networking under the Hood
● InfoQ Interview Ron Pressler
● State Of Loom Part 1 (Part 2)
● Helidon Nima
● Loom and Thread Fairness (Morling)
● Loom Comparison
● Structured Concurrency
● Mario Fusco: Game of Loom
Resources

A celebratory poem
In threads of fate, a loom converged,
A tapestry of change emerged,
From distant shores, the world was swirled,
As one, concurrent threads unfurled.
-- courtesy GPT-4

What can you do?
Test! Provide feedback!
Don't use in production (yet)!

Thank You!
… I'd love to take questions!

Looming Marvelous - Virtual Threads in Java Javaland.pdf

More Related Content

What's hot (20)

Similar to Looming Marvelous - Virtual Threads in Java Javaland.pdf (20)

More from jexp (20)

Recently uploaded (20)

Looming Marvelous - Virtual Threads in Java Javaland.pdf