Putting Java Streams Through Benchmarks
Having looked at the Java ExecutorService interface and its implementations, we now test how the Fork/Join framework and ThreadPerTaskExecutor cope under pressure.
Previously, I examined the Java ExecutorService interface and its implementations more closely, paying particular attention to the Fork/Join framework and ThreadPerTaskExecutor. This time, I want to go one step further and see how they cope under pressure. Put simply, I am going to run benchmarks – plenty of them.
Logic Under Benchmark
Let me begin by going through the logic on which the benchmarks are built, since it falls into two basic categories:
- based on the classic stream
- based on the Fork/Join approach
Classic Stream Logic
public static Map<Ip, Integer> groupByIncomingIp(Stream<String> requests,
LocalDateTime upperTimeBound, LocalDateTime lowerTimeBound) {
return requests
.map(line -> line.split(","))
.filter(words -> words.length == 3)
.map(words -> new Request(words[1], LocalDateTime.parse(words[2])))
.filter(request -> request.timestamp().isBefore(upperTimeBound) &&
request.timestamp().isAfter(lowerTimeBound))
.map(i -> new Ip(i.ip()))
.collect(groupingBy(i -> i, summingInt(i -> 1)));
}
In principle, this code takes a list of strings, transforms it, applies some filtering and grouping, and returns a map. The strings it receives look like this:
1,192.168.1.1,2023-10-29T17:33:33.647641574
Each one represents an event in which an IP address attempts to access a given server. The result maps every IP address to the number of access attempts within a given period, defined by a lower and an upper time boundary.
Fork/Join Logic
@Override
public Map<Ip, Integer> compute() {
if (data.size() >= THRESHOLD) {
Map<Ip, Integer> output = new HashMap<>();
ForkJoinTask
.invokeAll(createSubTasks())
.forEach(task -> task
.join()
.forEach((k, v) -> updateOutput(k, v, output))
);
return output;
}
return process();
}
private void updateOutput(Ip k, Integer v, Map<Ip, Integer> output) {
Integer currentValue = output.get(k);
if (currentValue == null) {
output.put(k, v);
} else {
output.replace(k, currentValue + v);
}
}
private List<ForkJoinDefinition> createSubTasks() {
int size = data.size();
int middle = size / 2;
return List.of(
new ForkJoinDefinition(new ArrayList<>(data.subList(0, middle)), now),
new ForkJoinDefinition(new ArrayList<>(data.subList(middle, size)), now)
);
}
private Map<Ip, Integer> process() {
return groupByIncomingIp(data.stream(), upperTimeBound, lowerTimeBound);
}
The one meaningful difference is that I keep dividing the dataset into smaller batches until it hits a given threshold, which defaults to 20. Only then do I begin the computations, which are identical to those of the classic stream logic described earlier – I rely on the groupByIncomingIp method.
JMH Setup
Every benchmark is written with the Java Microbenchmark Harness (JMH for short).
I ran the benchmarks with JMH version 1.37. All of them use the same setup: five warm-up iterations followed by twenty measurement iterations.
Two different modes are used: average time and throughput. With average time, JMH measures the mean execution time of the benchmarked code, and the result is given in milliseconds.
With throughput, JMH counts how many operations - complete executions of the code - fit into a given unit of time, here milliseconds. The result is given in ops per millisecond.
Expressed in JMH terms:
@Warmup(iterations = 5, time = 10, timeUnit = SECONDS)
@Measurement(iterations = 20, time = 10, timeUnit = SECONDS)
@BenchmarkMode({Mode.AverageTime, Mode.Throughput})
@OutputTimeUnit(MILLISECONDS)
@Fork(1)
@Threads(1)
In addition, every benchmark has its own State with a Benchmark scope that holds all the data and variables that benchmark requires.
Benchmark State
Classic Stream
Below you can find the base benchmark state for Classic Stream.
@State(Scope.Benchmark)
public class BenchmarkState {
@Param({"0"})
public int size;
public List<String> input;
public ClassicDefinition definitions;
public ForkJoinPool forkJoinPool_4;
public ForkJoinPool forkJoinPool_8;
public ForkJoinPool forkJoinPool_16;
public ForkJoinPool forkJoinPool_32;
private final LocalDateTime now = LocalDateTime.now();
@Setup(Level.Trial)
public void trialUp() {
input = new TestDataGen(now).generate(size);
definitions = new ClassicDefinition(now);
System.out.println(input.size());
}
@Setup(Level.Iteration)
public void up() {
forkJoinPool_4 = new ForkJoinPool(4);
forkJoinPool_8 = new ForkJoinPool(8);
forkJoinPool_16 = new ForkJoinPool(16);
forkJoinPool_32 = new ForkJoinPool(32);
}
@TearDown(Level.Iteration)
public void down() {
forkJoinPool_4.shutdown();
forkJoinPool_8.shutdown();
forkJoinPool_16.shutdown();
forkJoinPool_32.shutdown();
}
}
To start, I initialize every variable the benchmarks need. Except for the size parameter, which plays a special role here, the thread pools are used only within the benchmark.
The size parameter, by contrast, is a rather interesting mechanism provided by JMH. It lets you parametrize a given variable used during the benchmark. Later on, when we get to running the benchmarks, you will see how I made use of it.
For now, I rely on this parameter to generate the input dataset, which stays the same for the entire benchmark - so that the results are more repeatable.
Next comes the up method, which behaves much like @BeforeEach from the JUnit library.
It runs before every one of the 20 iterations of my benchmark and resets all the variables the benchmark uses. This way, each iteration begins with a clean state.
Finally, there is the down method, which behaves much like @AfterEach from the JUnit library.
It runs after every one of the 20 iterations of my benchmark and shuts down all the thread pools used in that iteration - mainly to deal with potential memory leaks.
Fork/Join
This is what the state for the Fork/Join version looks like.
@State(Scope.Benchmark)
public class ForkJoinState {
@Param({"0"})
public int size;
public List<String> input;
public ForkJoinPool forkJoinPool_4;
public ForkJoinPool forkJoinPool_8;
public ForkJoinPool forkJoinPool_16;
public ForkJoinPool forkJoinPool_32;
public final LocalDateTime now = LocalDateTime.now();
@Setup(Level.Trial)
public void trialUp() {
input = new TestDataGen(now).generate(size);
System.out.println(input.size());
}
@Setup(Level.Iteration)
public void up() {
forkJoinPool_4 = new ForkJoinPool(4);
forkJoinPool_8 = new ForkJoinPool(8);
forkJoinPool_16 = new ForkJoinPool(16);
forkJoinPool_32 = new ForkJoinPool(32);
}
@TearDown(Level.Iteration)
public void down() {
forkJoinPool_4.shutdown();
forkJoinPool_8.shutdown();
forkJoinPool_16.shutdown();
forkJoinPool_32.shutdown();
}
}
The setups for classic stream and Fork/Join hardly differ. The sole difference is that the definitions are created inside the benchmarks themselves rather than in the state, as is the case with the Classic approach.
This change stems from the way RecursiveTask works - task executions get memoized and stored - which could affect the overall benchmark results.
Benchmark Input
The core input of the benchmarks is a list of strings in this format
1,192.168.1.1,2023-10-29T17:33:33.647641574
or, described more generally
{ordering-number},{ip-like-string},{timestamp}
Five different input sizes are used:
- 100
- 1000
- 10000
- 100000
- 1000000
These sizes were not picked at random: I believe a range like this can show how well the solution scales and may reveal a performance bottleneck.
The benchmark setup is also highly flexible overall, so anyone who wants to add another size should be able to do so without much trouble.
Benchmark Setup
Classic Stream
The classic stream benchmark consists of just one class. The different sizes are handled at the State level.
public class ClassicStreamBenchmark extends BaseBenchmarkConfig {
@Benchmark
public void bench_sequential(SingleStreamState state, Blackhole bh) {
Map<String, Integer> map = state.definitions.sequentialStream(state.input);
bh.consume(map);
}
@Benchmark
public void bench_defaultParallelStream(SingleStreamState state, Blackhole bh) {
Map<String, Integer> map = state.definitions.defaultParallelStream(state.input);
bh.consume(map);
}
@Benchmark
public void bench_parallelStreamWithCustomForkJoinPool_4(SingleStreamState state, Blackhole bh) {
Map<String, Integer> map = state.definitions.parallelStreamWithCustomForkJoinPool(state.forkJoinPool_4, state.input);
bh.consume(map);
}
@Benchmark
public void bench_parallelStreamWithCustomForkJoinPool_8(SingleStreamState state, Blackhole bh) {
Map<String, Integer> map = state.definitions.parallelStreamWithCustomForkJoinPool(state.forkJoinPool_8, state.input);
bh.consume(map);
}
@Benchmark
public void bench_parallelStreamWithCustomForkJoinPool_16(SingleStreamState state, Blackhole bh) {
Map<String, Integer> map = state.definitions.parallelStreamWithCustomForkJoinPool(state.forkJoinPool_16, state.input);
bh.consume(map);
}
@Benchmark
public void bench_parallelStreamWithCustomForkJoinPool_32(SingleStreamState state, Blackhole bh) {
Map<String, Integer> map = state.definitions.parallelStreamWithCustomForkJoinPool(state.forkJoinPool_32, state.input);
bh.consume(map);
}
}
The same logic is benchmarked in six different setups:
- bench_sequential -> a basic benchmark using only a single sequential stream
- bench_defaultParallelStream -> a benchmark using Java’s default parallel stream through the .parallelStream() method of the
Streamclass, which in practice means thecommonPoolfrom ForkJoinPool with a parallelism of 19 (at least on my machine) - bench_parallelStreamWithCustomForkJoinPool_4 -> a custom
ForkJoinPoolwhose parallelism level is 4 - bench_parallelStreamWithCustomForkJoinPool_8 -> a custom
ForkJoinPoolwhose parallelism level is 8 - bench_parallelStreamWithCustomForkJoinPool_16 -> a custom
ForkJoinPoolwhose parallelism level is 16 - bench_parallelStreamWithCustomForkJoinPool_32 -> a custom
ForkJoinPoolwhose parallelism level is 32
For the classic stream logic, 6 different setups combined with 5 different input sizes give 30 unique benchmark combinations in total.
Fork/Join
public class ForkJoinBenchmark extends BaseBenchmarkConfig {
@Benchmark
public void bench(ForkJoinState state, Blackhole bh) {
Map<Ip, Integer> map = new ForkJoinDefinition(state.input, state.now).compute();
bh.consume(map);
}
@Benchmark
public void bench_customForkJoinPool_4(ForkJoinState state, Blackhole bh) {
ForkJoinDefinition forkJoinDefinition = new ForkJoinDefinition(state.input, state.now);
Map<Ip, Integer> map = state.forkJoinPool_4.invoke(forkJoinDefinition);
bh.consume(map);
}
@Benchmark
public void bench_customForkJoinPool_8(ForkJoinState state, Blackhole bh) {
ForkJoinDefinition forkJoinDefinition = new ForkJoinDefinition(state.input, state.now);
Map<Ip, Integer> map = state.forkJoinPool_8.invoke(forkJoinDefinition);
bh.consume(map);
}
@Benchmark
public void bench_customForkJoinPool_16(ForkJoinState state, Blackhole bh) {
ForkJoinDefinition forkJoinDefinition = new ForkJoinDefinition(state.input, state.now);
Map<Ip, Integer> map = state.forkJoinPool_16.invoke(forkJoinDefinition);
bh.consume(map);
}
@Benchmark
public void bench_customForkJoinPool_32(ForkJoinState state, Blackhole bh) {
ForkJoinDefinition forkJoinDefinition = new ForkJoinDefinition(state.input, state.now);
Map<Ip, Integer> map = state.forkJoinPool_32.invoke(forkJoinDefinition);
bh.consume(map);
}
}
The same logic is benchmarked in six different setups:
- bench -> a basic benchmark using only a single sequential stream
- bench_customForkJoinPool_4 -> a custom ForkJoinPool whose parallelism level is 4
- bench_customForkJoinPool_8 -> a custom ForkJoinPool whose parallelism level is 8
- bench_customForkJoinPool_16 -> a custom ForkJoinPool whose parallelism level is 16
- bench_customForkJoinPool_32 -> a custom ForkJoinPool whose parallelism level is 32
For the classic stream logic, 5 different setups combined with 5 different input sizes give 25 unique benchmark combinations in total.
In addition, both cases make use of the JMH Blackhole concept to “trick” the compiler’s dead code optimization. You can read more about Blackholes and when to use them here.
Benchmark Environment
Machine 1
The tests we ran on my Dell XPS, which has these parameters:
OS: Ubuntu 20.04.6 LTS
CPU: i9-12900HK × 20
Memory: 64 GB
JVM
openjdk version “21” 2023-09-19
OpenJDK Runtime Environment (build 21+35-2513)
OpenJDK 64-Bit Server VM (build 21+35-2513, mixed mode, sharing)
Machine 2
The tests we ran on my Lenovo Y700, which has these parameters:
OS: Ubuntu 20.04.6 LTS
CPU: i7-6700HQ × 8
Memory: 32 GB
JVM
openjdk version “21” 2023-09-19
OpenJDK Runtime Environment (build 21+35-2513)
OpenJDK 64-Bit Server VM (build 21+35-2513, mixed mode, sharing)
On both machines, every secondary or unimportant application was shut down. I aimed to keep the runtime system as clean as I could so as not to introduce any unwanted performance overhead. Still, overall performance may be different on a bare Ubuntu server or when running inside a container.
Benchmark Report
The benchmark results are saved in .csv files, which live in the GitHub repository under the reports directory. To make downloading the reports easier, there is also a separate .zip file called reports.zip containing all the .csv data files.
The report directories are organised by size, plus three special reports covering all input sizes:
- report_classic - every input size for the classic stream
- report_forkjoin - every input size for the fork/join stream
- report_whole - every input size for both the classic and fork/join streams
Each of the report directories above holds 3 separate files:
- averagetime.csv - results of the average time mode benchmarks
- throughput.csv - results of the throughput mode benchmarks
- total.csv - the results of both modes combined
These reports come in two formats - averagetime.csv and throughput.csv share one, while total.csv uses another. Let’s refer to them as the modes format and the total format.
The modes report has eight columns:
- Label - the benchmark’s name
- Input Size - the size of the benchmark input
- Threads - how many threads the benchmark used, from the set 1,4,7,8,16,19,32
- Mode - the benchmark mode, either average time or throughput
- Cnt - the number of benchmark iterations, which should always be 20
- Score - the actual benchmark result
- Score Mean Error - the benchmark’s measurement error
- Units - the benchmark’s units, either ms/op (for average time) or ops/ms (for throughput**)**
The total report has 10 columns:
- Label - the benchmark’s name
- Input Size - the size of the benchmark input
- Threads - how many threads the benchmark used, from the set 1,4,7,8,16,19,32
- Cnt - the number of benchmark iterations, which should always be 20
- AvgTimeScore - the actual benchmark result in average time mode
- AvgTimeMeanError - the benchmark’s measurement error in average time mode
- AvgUnits - the benchmark’s units in average time mode, in ms/op
- ThroughputScore - the actual benchmark result
- ThroughputMeanError - the benchmark’s measurement error in throughput mode
- ThroughputUnits - the benchmark’s units in throughput mode, in ops/ms
Results Analysis
Assumptions
Baseline
The general results and insights I present are based on size 10000 – meaning I use the .csv files from the report_10000 directory.
I picked this particular data size for two main reasons.
- The execution time is long enough to reveal any differences between the setups.
- In my view, data sizes 100 and 1000 are too small to expose certain performance bottlenecks
For this reason, I believe a detailed analysis of this data size will be the most valuable.
Naturally, the other sizes will get an overview of their results as well, though not as detailed as this one, unless I come across anomalies – compared with how size 10000 behaves.
A Word On Fork/Join Native Approach
Given the code currently being benchmarked, the Fork/Join benchmarks carry some performance overhead.
Since the fork join benchmark logic depends heavily on splitting the input dataset, at some point all the partial results have to be merged into one coherent output. This step is absent from the regular benchmarks, so it is essential to understand properly how it affects overall performance.
Please keep this in mind.
Analysis
Machine 1 (20 cores)
As shown above, on machine 1 the best overall result for an input of 10 thousand comes from the defaultParallelStream versions.
Among the ClassicStream based benchmarks, bench_defaultParallelStream delivers by far the best result. It stays in first place even after accounting for possible measurement error.
The ForkJoinPool setups with parallelism 32 and 16 and deliver worse results. In one sense, this is surprising - with parallelism 32 I would have expected a better score than with the default pool (parallelism 19). In another, it is not - parallelism 16 performs worse than both parallelism 19 and 32.
Since Machine 1 has 20 CPU threads, parallelism 32 is not high enough to show the performance drop caused by having too many threads. You would, however, see such behavior on Machine 2. My guess is that to reproduce it on Machine 1, parallelism would need to be set to 64 or higher.
Interestingly, bench_defaultParallelStream does not seem to stay on top for the larger input sizes of 100k and one million. There, the best performance comes from bench_parallelStreamWithCustomForkJoinPool_16, which may suggest that a reasonably lower parallelism can ultimately be a good choice.
The Fork/Join-based implementation is clearly slower than the default parallel stream implementation, performing roughly 10 % worse. The same pattern shows up for the other sizes too. This confirms my earlier assumption that merging the smaller pieces of a split dataset has a noticeable cost.
Unsurprisingly, the single-threaded approach scores worst, about 5 times slower than the best result. That is to be expected, since I treat the single-threaded benchmark as a sort of baseline. I want to see how much its execution time can be improved, and an average execution time 5 times better in the best case looks like a good result.
The score mean error is extremely small. Even at its worst (the highest error), it stays within 1,5% of the corresponding score - the result for ClassicStreamBenchmark.bench_parallelStreamWithCustomForkJoinPool_4.
In the remaining cases, it ranges from 0,1 % to 0,7 % of the total score.
For sizes above 10 thousand, the ranking of results appears not to change.
Machine 2 (8 cores)
Just as on Machine 1, the top score again goes to bench_defaultParallelStream. And again, it stays in first place even when possible measurement error is taken into account. Nothing particularly noteworthy.
What is noteworthy, though, is that on Machine 2 the order of the top 3 positions shifts considerably for larger input sizes. For inputs from 100 to 10 000, the behavior is fairly similar, with bench_defaultParallelStream in 1st place and bench_parallelStreamWithCustomForkJoinPool_8 close behind.
For inputs of 100 000 and 1 000 000, by contrast, bench_parallelStreamWithCustomForkJoinPool_8 takes first place, with bench_parallelStreamWithCustomForkJoinPool_32 right after it. Meanwhile, bench_defaultParallelStream drops to 4th and 3rd place
Another interesting aspect of Machine 2 is that for smaller input sizes, parallelism 32 lands far from the top. This drop in performance may be due to having too many threads relative to the 8 CPU threads the machine has in total.
Even so, for inputs of 100 000 and 1 000 000, ForkJoinPool with parallelism 32 takes second place, which may suggest that over longer time spans having too many threads is not an issue.
A few other aspects that closely mirror Machine 1’s behavior are omitted here and covered below.
Common points
Some observations hold true for both machines:
- My ForkJoinNative (“naive”)-based benchmarks produce clearly worse results, about 10% on both machines, than the default parallel stream versions or even the ones using customForkJoinPool.
Naturally, part of the reason is that they have not been optimized at all, and there are likely some easy performance wins to be had. That is why I strongly advise getting to know the Fork/Join framework well before taking its implementations to production. - The gap in time between first and third place is tiny - under a millisecond. This can make these benchmarks hard to reproduce in any meaningful way. With such a small gap, the distribution of results can easily vary from one benchmark run to another.
- The mean Error of the scores is also tiny, at most 2% of the total score in the worse cases - and usually below 1%. An error this low may point to two things. First, the benchmarks are reliable, since the results cluster around a certain point; had there been anomalies, the error would be larger. Second, JMH does a good job of taking measurements.
- Throughput and average time modes do not produce any fundamentally different results. A benchmark that did well in average time mode also did well in throughput mode.
Those are all the differences and similarities I spotted in the report files. If anything else catches your eye as interesting, feel free to mention it in the comments below.
Summary
Before we part ways for today, there is one more very important thing I want to point out.
JAVA IS NOT SLUGGISH
Processing a list of one million elements, with all the potential JMH overhead and on a single thread, takes 560 milliseconds (Machine 1) and 1142 milliseconds (Machine 2). No special optimizations or tricks were applied – just a plain default JVM.
The best overall time for processing one million elements on Machine 1 was 88 milliseconds, achieved by ClassicStreamBenchmark.bench_parallelStreamWithCustomForkJoinPool_16. On Machine 2, it was 321 milliseconds, achieved by ClassicStreamBenchmark.bench_parallelStreamWithCustomForkJoinPool_8.
Neither result may match C/C++-based solutions, but in my view the comparative simplicity and expressiveness of this approach make it very appealing.
All in all, it makes a pretty nice complement to Java’s one billion rows challenge.
You can run the same benchmarks to compare my results with how they behave on your own machine.
Also, to make downloading the reposts easier, there is a separate .zip file called reports.zip containing all the .csv data files.
And remember: Java is not sluggish.
Thanks for your time.