What does blocking mean in Loom?
Project Loom brings Virtual Threads to the Java runtime, shipping in JDK 21. To reach its performance targets, Loom's runtime takes care of blocking operations.
Project Loom brings the idea of Virtual Threads into the Java runtime, and in September it will ship as a stable feature of JDK 21. The goal of Project Loom is to merge the performance advantages of asynchronous programming with the ease of a direct, “synchronous” style of writing code.
To reach those performance targets, Loom’s runtime must treat every blocking operation in a particular way. Let’s look into how this special treatment works and whether programming with Loom involves any corner cases.
Before we start, it is worth clarifying what blocking means here: it is a state of a program’s thread in which the thread does no useful work (it doesn’t use the CPU) and instead waits for some external condition to be met. Typical examples are waiting on a semaphore or waiting for data to arrive in a socket.
Carrier & virtual threads
To begin, here is a brief overview of Loom’s key concepts. At the foundation there are platform threads, also called kernel threads. Java has had these threads for a long time; so far, every running Thread instance has mapped to exactly one kernel thread. Such threads are heavy-weight, and both creating them and switching between them is costly. They are a limited resource that must be managed with care, for instance through a thread pool.
Virtual threads are a new concept brought in by Loom. They are lightweight and inexpensive to create, both memory-wise and in terms of the time it takes to switch contexts. In a JDK where virtual threads are enabled, a Thread instance may stand for either a platform thread or a virtual one. The API stays identical, but the cost of running each kind differs considerably.
Under the hood, the JVM+Loom runtime maintains a pool of platform threads, known as carrier threads, onto which virtual threads are multiplexed. In other words, a handful of platform threads execute a large number of virtual threads. Each time a virtual thread calls a blocking operation, it should be “set aside” until the condition it waits for is satisfied, so that a different virtual thread can run on the carrier thread that has just been released.
This idea isn’t new: various reactive libraries have implemented it with thread pools and non-blocking I/O. Here, though, it happens at the runtime level rather than in a library, which gives us far cleaner syntax and meaningful stack traces, and spares us the virality of things like the Future wrapper.
That’s as far as we’ll go with the Loom introduction; for a more thorough exploration, I recommend The Ultimate guide to Virtual Threads on the Rock the JVM blog.
A key point is that for the system to keep making steady progress (when many virtual threads are in use), carrier threads must be released often enough for virtual threads to get scheduled on them. That’s why I/O-heavy systems should benefit the most, whereas CPU-heavy applications will gain little from adopting Loom.
All of this assumes that every blocking operation releases the carrier thread so that a different virtual thread can take its place. But does it really?
Retrofitting blocking
As noted earlier, implementing virtual threads largely meant that Project Loom had to retrofit existing blocking operations to make them aware of virtual threads. As a result, when such operations are called, they release the carrier thread, letting other virtual threads continue.
For instance, when code blocks on a semaphore, a lock, or some other Java concurrency primitive, the underlying carrier thread isn’t blocked. Instead, the runtime is signalled to capture the continuation of the current virtual thread, place it in a waiting queue, and resume it once the condition that caused the blocking is resolved.
Likewise, Thread.sleep now blocks only the virtual thread and leaves the carrier thread free. Consequently, the following code finishes in 4 seconds, despite only around 15 platform threads being started (the size of the carrier thread pool that runs virtual threads depends on how many cores the JVM has at its disposal):
var e = Executors.newVirtualThreadPerTaskExecutor();
for (int i = 0; i < 1000; i++) {
e.submit(() -> { sleep(4000); });
}
e.shutdown();
e.awaitTermination(1, TimeUnit.DAYS);
Synchronization problems
Not every Java construct, however, has been retrofitted in this manner. Among those that haven’t are synchronized methods and code blocks. When they are used, the virtual thread gets pinned to its carrier thread. While a thread is pinned, any blocking operation blocks the underlying carrier thread – exactly as it did before Loom.
Consider this example:
var e = Executors.newVirtualThreadPerTaskExecutor();
for (int i = 0; i < 1000; i++) {
e.submit(() -> { new Test().test(); });
}
e.shutdown();
e.awaitTermination(1, TimeUnit.DAYS);
class Test {
synchronized void test() {
sleep(4000);
}
}
Observe that every submitted task instantiates a new Test, so a different monitor is used for each call of the synchronized void test() method. Still, since we rely on synchronized, the thread gets pinned to the carrier thread. With 5 threads in the carrier pool, only five tasks can run at once (because they block the whole pool). As a result, even though this code differs only slightly from the previous example, it needs 1000/5*4 = 800 seconds to finish.
The fix is to replace the synchronized keyword with locks:
var e = Executors.newVirtualThreadPerTaskExecutor();
for (int i = 0; i < 1000; i++) {
e.submit(() -> { new Test().test(); });
}
e.shutdown();
e.awaitTermination(1, TimeUnit.DAYS);
class Test {
private ReentrantLock lock = new ReentrantLock();
void test() {
lock.tryLock();
try {
sleep(4000);
} finally {
lock.unlock();
}
}
}
With Loom, this again finishes in 4 seconds.
Swapping synchronized blocks for locks inside the JDK (wherever feasible) is another part of Project Loom’s scope and of what ships in JDK 21. Various Java and JVM libraries have already made, or are currently making, the same kind of changes (JDBC drivers, for example). Application code relying on synchronized, however, will require additional attention.
Sockets
Are there other areas, like synchronized, that lead to thread pinning and could therefore hurt our application’s performance?
Fortunately, one of the most widespread sources of I/O – network communication over TCP sockets – is entirely Loom-aware. That is hardly surprising: sockets can be handled in a non-blocking fashion (either fully non-blocking or with a single “manager” thread coordinating the execution of many I/O tasks) through mechanisms like select() or epoll().
This is precisely what Project Loom relies on internally to offer a socket implementation that plays well with virtual threads. The non-blocking I/O details stay hidden, and we work with a familiar, synchronous API. A complete example that uses a java.net.Socket directly would be quite long, so imagine one that sends several requests concurrently to a server that replies after 3 seconds.
On closer inspection, you’ll notice InputStream.read calls wrapped in a BufferedReader that reads from the socket’s input. This is the blocking call that suspends the virtual thread. With Loom, the test finishes in 3 seconds, although only 16 platform threads are ever started in the whole JVM and 50 requests run concurrently.
The picture changes, however, for datagram sockets (which use the UDP protocol). Again, we have a server that answers with a UDP packet after 3 seconds, but when you run the client, you’ll find that DatagramSocket.receive pins the thread, so at any given moment only a few clients (at most as many as the carrier thread pool size) can make progress. Consequently, the whole test runs longer.
So far we’ve identified at least two causes of thread pinning: synchronized blocks and datagram sockets. Are there others?
Files & DNS
There are – other APIs also pin threads and block the carrier thread, namely every file operation (reading from a FileInputStream, writing to a file, listing directories and so on), as well as resolving domain names into IP addresses with InetSocketAddress.
This may seem alarming, but Loom does apply some countermeasures. Looking into the source code of FileInputStream, InetSocketAddress or DatagramSocket, you’ll spot uses of the jdk.internal.misc.Blocker class. Calls to its begin()/end() methods wrap every call that blocks the carrier thread.
Its JavaDoc explains what it’s for:
/**
* Defines static methods to mark the beginning and end of a possibly blocking
* operation. The methods are intended to be used with try-finally as follows:
* {@snippet lang=java :
* long comp = Blocker.begin();
* try {
* // blocking operation
* } finally {
* Blocker.end(comp);
* }
* }
* If invoked from a virtual thread and the underlying carrier thread is a
* CarrierThread then the code in the block runs as if it were in run in
* ForkJoinPool.ManagedBlocker. This means the pool can be expanded to support
* additional parallelism during the blocking operation.
*/
public class Blocker
Put differently, when a blocking operation occurs, the carrier thread pool may be expanded to compensate for the resulting thread pinning. A fresh carrier thread can be started, capable of running virtual threads.
Side note: checking where the
Blockerclass is used is also a handy way to find out which APIs pin threads in a given JDK version.
Discovering pinned threads and carrier pool configuration
If thread pinning makes you uneasy, you’re in good company – it’s certainly something to keep an eye on. Fortunately, the JVM can report every occurrence of it when you pass the -Djdk.tracePinnedThreads=short flag (a full option is available as well, which prints the complete stack trace).
This produces messages like:
Thread[#91,ForkJoinPool-1-worker-7,5,CarrierThreads]
java.base/sun.nio.ch.DatagramSocketAdaptor.receive(DatagramSocketAdaptor.java:241) <== monitors:1
Additionally, the initial and maximum size of the carrier thread pool can be tuned with the jdk.virtualThreadScheduler.parallelism, jdk.virtualThreadScheduler.maxPoolSize and jdk.virtualThreadScheduler.minRunnable configuration options. They map directly onto constructor arguments of the ForkJoinPool.
This won’t prevent thread pinning, but it does let you detect when it occurs and, where necessary, rework the offending code paths.
Are we doomed to blocking?
To some extent, yes – certain operations are blocking by nature because of the way our operating systems are built.
Take page faults: if our application accesses memory that has been swapped out to disk, a page fault occurs. The operating system has to load the memory content from disk before our application can proceed. During that load, the thread touching the page is blocked. Neither the JVM nor any other runtime has any say in this. Such low-level memory management is entirely the kernel’s job and is fully invisible to the application. So we can never eliminate blocking completely, even if it only takes the form of page faults.
Next, what about files? From the kernel’s point of view, reading from a socket can block, because the socket may not have data yet (it may not be “ready”). Reading from a socket may mean waiting for data to come in over the network. Files are a different story, since they are read from block devices available locally. The data is always there; at most it has to be copied from disk into memory.
In a sense, as far as the kernel is concerned, file operations never block the way socket operations do. Reading a file resembles a page fault to some degree. For that reason, every kernel API for file access is, in the end, blocking (in the sense defined at the start).
select, which handled our socket problems, is of no use with descriptors for local files: they are reported as “ready” straight away. And epoll/aio can’t be used with local files at all. Neither Java nor any other runtime can turn these calls into non-blocking ones.
Will io_uring save the day?
Perhaps you’ve come across io_uring: a recent kernel addition designed specifically to offer a uniform asynchronous API for every kind of I/O operation, above all operations on local files. Couldn’t Loom make use of it?
It could, and most likely it will (probably just for local files, since io_uring’s performance advantage over epoll isn’t consistent, and the implementation itself regularly suffers from security vulnerabilities). Still, that merely moves the problem elsewhere rather than solving it completely.
The application sees a non-blocking, asynchronous API for accessing files. However, peeking at what io_uring does internally reveals that it maintains a thread pool for blocking operations, such as those on local files. So rather than the JVM running compensating threads, we end up with threads started and managed by io_uring.
This affects configuration. To limit the number of kernel threads your application uses, you’ll have to configure both the JVM’s carrier thread pool and io_uring, capping the maximum number of threads the latter starts. Fortunately, a great article explains exactly how to do that.
Do we want a uniform I/O interface?
It’s now clear that I/O operations aren’t all alike: the two most frequent kinds, working with network sockets and with local files, behave differently at the kernel level. Efforts like io_uring may aim for a uniform interface, but underneath, one kind of I/O is non-blocking, whereas the other is blocking and will run on a thread pool.
Given that the two kinds of I/O operations have different characteristics, is a uniform interface sensible? Isn’t that a leaky abstraction?
Daniel Spiewak started an interesting online discussion on precisely this subject. Daniel’s argument is that since files and sockets block differently, this difference should not be concealed behind an abstraction layer like io_uring or Loom’s virtual threads, but rather made visible to the developer. The reason is that they should be used in different patterns, and every blocking call should be batched & guarded by a gateway, for example a semaphore or a queue.
I, on the other hand, would argue that non-blocking I/O, as with sockets, still isn’t free. It may be cheaper than blocking I/O, but our code should gate the use of every kind of I/O appropriately. The concrete concurrency limits we set for each kind of operation may vary, but they should exist nonetheless.
Naturally, it’s essential to know up front how each kind of I/O operation behaves. So Daniel has a point that such a uniform interface can complicate matters: it isn’t obvious at a glance (the type system offers no help, for example) how a particular operation will behave at runtime.
How much control do we want?
Daniel also rightly notes that the blocking/non-blocking distinction really only matters under high load – but that may well be the same scale at which async or Loom/io_uring matter in the first place. And at that scale you may want to know precisely what calling a particular method entails.
He closes with a philosophical question about how strictly the runtime should honour the user’s requests. Should we be able to say “run this I/O operation without blocking” – and, if that can’t be done (say, because a local file is involved), should the whole call fail? Or should the runtime always try its best to fulfil our requests, whatever the cost?
One possible answer is a hierarchical capability system – offering both fine-grained BlockingIO and NonBlockingIO capabilities and coarser IO ones. The type system could then guide us towards handling each case correctly, covering both its failure modes and its runtime characteristics.
Summing up
Here are a few things we’ve learned on the way:
- The difference between carrier and virtual threads, and the fact that Loom runs many virtual threads on a small pool of carrier threads.
- Part of Loom is retrofitting blocking operations so that they become aware of virtual threads.
- Concurrency primitives such as locks and semaphores, as well as socket-based networking, have been migrated to work correctly with virtual threads.
- That doesn’t hold for every blocking API in the JDK, though. Whenever a carrier thread gets blocked, we’re dealing with thread pinning.
- Thread pinning happens with the
synchronizedkeyword, as well as during file I/O, when using datagram sockets, and when resolving domain names. - There are ways to avoid thread pinning: replacing
synchronizedwith locks, and starting compensating carrier threads. The JDK does the latter on its own (up to the configured limit) whenever an I/O operation that blocks the carrier thread is called. - Certain operations, such as file I/O, are blocking by nature at the kernel level, and Loom cannot make them non-blocking.
io_uringsolves the problem only partially, since internally it relies on a thread pool to run blocking operations (such as file I/O).
We ended with an open question: since no runtime or API can deliver truly universal non-blocking I/O, is it wise to have an abstraction that offers one uniform interface for every blocking and non-blocking operation?
The answer may well be “it depends”. Past a certain scale, these details begin to matter. At smaller scales, they may be irrelevant. Abstractions like Loom or io_uring are leaky and can mislead. Perhaps some guidance from the type system would help. Or perhaps this information could be communicated differently? Finally, we may want a way to tell our runtimes to fail when an I/O operation cannot be executed in a particular manner.
Loom moves the JVM forward considerably and meets its performance goals while simplifying the programming model; yet we can’t rely on it blindly to eliminate every source of kernel thread blocking from our applications. It may even introduce new performance problems into our applications while fixing others.
What’s your take on this – should blocking and non-blocking operations be told apart? And if so, how – at the type level, through a naming convention, or some other way? What configuration should our runtimes offer so that we know for sure how a particular I/O operation will run?