The first version of Takipi’s recent blog posting on 'How Misusing Streams Can Make Your Code 5 Times Slower' met with criticism about how benchmarking the performance degradation on Java 8 Streams was conducted, which they were quick to rectify with an optimized version of the original benchmark and corrected result attributions.
With the help of Streams and Lambdas in Java 8, Java developers can utilize functional programming style, which is different than programming with traditional For-Loops and Iterators. (InfoQ has previously pitted Java 8’s functional style against imperative style programming in Java.)
Takipi’s blog post tested Java 8’s functional programming features using an example of finding a maximum value in an ArrayList. The test compared functional vs. imperative programming style by implementing Iterators, For-loops, For-Each loops, Stream, parallel Stream, and Lambdas (with and without For-Each). The first iteration of this benchmarking failed to employ a few basic JIT compiler optimizations and in some cases a few optimizations were only employed for some of the test cases while others didn’t reap the benefits of those optimizations.
Following the community critiques, Takipi quickly revised their benchmark (built on JMH - the Java Microbenchmarking Harness). The first major change in the revised benchmark was removal of the volatile keyword for the ‘integers’ field as shown below:
- volatile List<Integer> integers = null;
+ List<Integer> integers = null;
As Oracle's Java performance engineer Sergey Kuksenko pointed out, the above change helped the JIT compiler to employ the ‘range check elimination’ optimization.
The second major change was the elimination of auto-boxing, since the benchmark would run into auto-boxing issues for Streams. The changes were made at multiple places in the benchmark code. One such change is shown in the code snippet below:
- Optional<Integer> max = integers.stream().reduce(Integer::max);
- return max.get();
+ return integers.stream().mapToInt(Integer::intValue).reduce(Integer.MIN_VALUE, Integer::max);
The revised benchmarking highlighted that parallel Stream had a slight edge over the other test cases; but overall the imperative programming style group of tests wins over the functional programming style group when the use case is as narrow as iterating through an ArrayList to find the max value.
InfoQ contacted the blog author, Alex Zhitnitsky and he mentioned:
There's a lot of excitement around the new Java 8 features, which is great, BUT many developers still misuse it. When benchmarking the use cases, we wanted to go for a non-optimized benchmark, since in real life day to day usage, many developers use these features out of the box.The post shows a specific use case that favors loops, compared with a sloppy yet short/intuitive/quick implementation with streams. So for example:
integers.stream().reduce(Integer::max);
versus
integers.stream().mapToInt(Integer::intValue).reduce(Integer.MIN_VALUE, Integer::max);
The second implementation would be the right way to go by the optimized benchmark, although it's easy to let mapToInt slip and create an auto-boxing issue with the shorter implementation. Both of the benchmarks are correct, since they both measure legitimate implementations - Even though the first version doesn't contain the optimizations (which can be non-intuitive, and longer, like in this mapToInt example)
The takeaway from this benchmarking exercise is that it’s really important to know what you are benchmarking and to profile and compare the generated code especially when employing micro-benchmarks.