Shenandoah is a low-latency garbage collector that enables Java applications to operate quickly without changes. The feature was first introduced upstream in JDK 12 and has since been backported to the long-term support JDK 11 that is used by approximately twenty to twenty five percent of surveyed Java users. This change to backport the garbage collector into the upstream OpenJDK11 repository gives Java vendors like Azul, Adoptium, and Liberica the ability to provide the feature to their users.
The consideration to add Shenandoah to the upstream repository posed a challenge of introducing a major feature into a version of Java that had already locked its feature-set. As more than a simple bug-fix, this major change created an opportunity for something to go wrong with users who simply expect to get a new JDK update that is basically the same as what they had before but with fixes. The Red Hat engineering team focused on these challenges to introduce Shenandoah in a safe manner.
The difference in the upstream repository and downstream distributions stems from the fact that OpenJDK is a community around source code, not an executable binary. The different Java vendors utilize the upstream repository and codebase to build their own binaries that different companies can install and use. For example, users of CentOS commonly use Red Hat's build of OpenJDK that has already featured Shenandoah for a long time.
InfoQ sat down with Roman Kennke and Aleksey Shipilev from Red Hat's Shenandoah team to ask some questions about how this garbage collector was created and what diligence went into adding it to the upstream JDK 11 codebase. Kennke is a principal software engineer and owner of JEP-304 (garbage collector interface), and Shipilev is also a principal software engineer and original author of Java Micro-Harness (JMH).
InfoQ: Can you give us a brief history on the introduction of Shenandoah? How long has it been under development?
Roman Kennke: The Shenandoah GC has been developed by Red Hat since JDK 8 timeframe (2013), and was first released in the JDK 8 builds in Red Hat Enterprise Linux 7.4 in 2017. When JDK 11 was released in 2018, Shenandoah was also released in corresponding JDK 11 packages starting in RHEL 7.6. Shenandoah GC was integrated in upstream OpenJDK in 2018 and first released as such starting with JDK 12. During the last couple of months and years, it has seen continuous improvements and releases in JDK 13, JDK 14 and recently JDK 15.
However, for many users, JDK 11 remains the most important release, because most JDK vendors treat it as an LTS - long-term-support - release. Especially conservative users are only now adopting JDK 11 (usually coming from JDK 8). Those users couldn't so far use the advantages that Shenandoah GC provided. We - the JDK team - received many requests from users and JDK vendors to put Shenandoah GC into JDK 11u upstream (remember, Red Hat has been shipping it in their own builds since 2018), so that it can be provided by all vendors who wish to build and ship it. We started that process early in 2019 and since July 2020, Shenandoah GC is integrated in the upstream source code of JDK 11u, and is expected to be released in mid-October 2020. It is not enabled to be built by default, but vendors or users who wish to build it can do so by the flip of a switch (--with-jvm-features=shenandoahgc).
InfoQ: How has the adoption and feedback from the community been with regards toShenandoah?
Kennke: Interestingly (or maybe not), adoption so far did not come so much from official (upstream) releases JDK 12 and later. We are getting most feedback from people who are using it in JDK 8 or JDK 11 (usually either Red Hat's builds - or derivatives -, or our own provided nightly builds, sometimes even self-built binaries). People seem to be using it for a wide range of purposes: from running their favorite IDE, running their huge old application server, in-memory databases. For example, I run my own IDE on Shenandoah, using IntelliJ for Java and CLion for C++. We also see users who are using it to run their cloud services in containers, which may be somewhat surprising because Shenandoah's original target-audience was huge, 100s of GB heap JVM instances. It turns out that short pause times are not only useful in huge settings, but also when resources (CPU, memory) are very constrained. Try running a JVM with default GC on a Raspberry Pi, then even the normal 5ms GC pause becomes 100s of ms or longer. Similar considerations apply for cloud deployments. Shenandoah performs very well in such environments for several reasons: it can be easily tuned to uncommit memory aggressively (compact mode -XX:ShenandoahGCHeuristics=compact), it is using relatively few auxiliary data structures which makes for native memory footprint comparable with the good old Serial GC, and, as mentioned earlier, short pause times remain short when CPU is constrained.
InfoQ: What were the possible risks in introducing Shenandoah into the upstream JDK 11 distribution?
Kennke: We got a lot of push-back (and rightly so) for risking to break *anything*. In particular, the high risk here has been that if anybody builds JDK11u with Shenandoah GC disabled, some Shenandoah code paths would still 'leak' out into shared code in Hotspot, and break something for unsuspecting who are not even using Shenandoah, and who not even asked for it. So we needed to make sure that this absolutely did not happen. Our ideal goal was that the resulting binary of a Shenandoah-less build would not have *any* traces of Shenandoah inside it, and we wanted to prove it, and thus prove that the inclusion of Shenandoah GC into JDK11u would not break any Shenandoah-less builds. Similarly, a Shenandoah-enabled build must not break anything either when running with a different GC.
InfoQ: What are the different tools and techniques that you used to generate that proof?
Kennke: First and foremost we have been testing Shenandoah GC in JDK11 thoroughly countless times. We have been shipping it to customers of Red Hat Enterprise Linux for several years after all, and want to sleep at night.
But we are going much further than this: we essentially wanted to isolate Shenandoah GC code as much as possible from the rest of the Hotspot JVM. Most of this isolation has been achieved during development of JDK9, 10 and 11, as part of the work for JEP 304 (Garbage Collector Interface), a collaboration between Red Hat and Oracle. If you compare JDK8 and JDK11 code bases and look for GC specific code, you will find it sprinkled all over the Hotspot source code in JDK8, and for the most part a nice clean interface for it in JDK11 and later. However, when JDK11 was originally released in 2018, this GC interface was not quite up to cover all of Shenandoah GC. (This was the main reason why Shenandoah GC had to be delayed to JDK12.) Which means that we still had to have a few cases where Shenandoah-specific code had to be inserted in shared code paths.
We have been very careful to guard all such inclusions with runtime-checks ( e.g. if (UseShenandoahGC) { … } ), and build-time checks (e.g. #if INCLUDE_SHENANDOAHGC ...). It was relatively easy to see that when looking at the source code patch, it would not build the enclosed code path when Shenandoah is disabled at build-time, and would not run that code path when it is built-in, but not active at run-time. However, it was not enough; we wanted to prove, on the level of the generated binary (the libjvm.so), that when Shenandoah is disabled from the build, that the resulting binary would be byte-by-byte *identical*.
In order to prove binary-identity between previous JDK11u builds and new JDK11u builds with Shenandoah GC disabled, I wrote a script that builds both a 'clean' (i.e. no Shenandoah changes at all) and 'patched' (JDK11u with Shenandoah, but excluded from the build), and compare the resulting libjvm.so byte by byte, and report any difference. It did find differences from two major and not Shenandoah related sources: line-numbers in debug-information (which is used to step through code when running in a native-code debugger like GDB), and time-stamps (which are used, for example, to report when the libjvm.so was built, in java --version or in hs_err files when the JVM crashes). Those are normally included by compiler built-in macros like __LINE__. It was possible to patch the build to not include line-numbers and timestamps (or rather, always include '0' in place of line-numbers and timestamps). After this we did indeed find a number of places where Shenandoah code leaked out into non-Shenandoah builds. For example, the C2 JIT compiler would generate some code at build-time that is used by the JIT back-end, and this code-generation emitted Shenandoah code. It was certainly worth doing this exercise in cleanliness, because it uncovered a couple of rather hidden paths where Shenandoah code crept into non-Shenandoah builds, and those paths were fixed.
InfoQ: What are some of the files inside a JDK distribution that interact with Shenandoah and other GCs, and what do they do?
Kenkke: GCs in OpenJDK/Hotspot are all written in C++. Most of GC code is in the directory src/hotspot/share/gc/; there you'll find the shared subdirectory which contains code that is shared between GCs and most importantly the GC interface, in other words, the set of C++ classes that a GC is expected to implement (e.g. CollectedHeap, BarrierSet, just to name a few most important ones). Implementations can be found in so-named subdirectories, e.g. src/hotspot/shared/gc/shenandoah for Shenandoah GC. The GC naturally interacts with the rest of the Hotspot runtime, but also with the code generation for the interpreter as well as C1 and C2 compilers. The GC specific parts of the latter two can be found in c1/ and c2/ sub-directories in each CG, e.g. src/hotspot/share/gc/shenandoah/c2 for the C2 JIT parts of Shenandoah GC. There is also platform-specific code for each GC, most importantly the interpreter-generator code. Those parts are located under src/hotspot/$ARCH/gc/$GC, e.g. src/hotspot/x86/gc/shenandoah for the X86-specific parts of ShenandoahGC.
Aleksey Shipilev: BarrierSets are a major facet of the GC interface. GCs need to interact with C++ runtime that wants access to the Java heap sometimes, and the application code that normally works with Java heap. Interaction with C++ runtime means every GC needs to implement the BarrierSet subclass. Application code comes in many forms: code executed by the template interpreter, code generated by C1 and code generated by C2.
The template interpreter requires writing out the GC barriers in high-level assembly, which also makes it platform-specific. Therefore, most GC implementations have BarrierSet, BarrierSetC1, BarrierSetC2, BarrierSetAssembler_x86, BarrierSetAssembler_aarch64, …, etc. This also explains what work do you need to do to support Graal with Shenandoah: not-yet-existent BarrierSetGraal.
InfoQ: There are a few known performance benchmarks: SpecJBB, Renaissance, and DaCapo, among others. How does a GC team use any of these benchmarks and what else do you use?
Kennke: We are running those benchmarks on a regular basis, and watch out for possible regressions there. Sometimes, especially when working on particular performance issues, we isolate certain parts of (bad) behaviour, try to amplify it and make a small microbenchmark out of it. This is easier to work with and most is covered on the OpenJDK Shenandoah page.
InfoQ: JDK Mission Control won the recent March-Madness bracket on feature popularity. How does Shenandoah interact with it?
Kenkke: Shenandoah emits relevant GC events and information like other GCs; that is, for example, GC heap occupancy, GC events for different phases, etc.
InfoQ: How do GCs identify objects? Do they leverage points like System.identityHashcode or something else?
Kennke: The GC is not only responsible for cleaning up garbage objects. Garbage collector is a misnomer. It should be called 'memory manager': the Java heap is basically under full control of the GC; the GC handles allocation *and* deallocation of objects. (Aleksey: because of this &quo;Epsilon GC&quo; exists). The tracing GC finds objects by doing a graph traversal: it starts from so-called GC-roots (e.g. reference variables on thread-stacks), and from those 'initial' objects, follows through references through the whole object graph to find all live objects.
This works with techniques like tri-color marking. If you want to learn more about this, I'd strongly recommend the great Garbage Collection Handbook.
InfoQ: What impact do you foresee with upcoming projects like Valhalla?
Shipilev: I think we can say here that GCs are working on per-object basis, and so many overheads are dependent on the number of objects the GC needs to work with. For example, marking includes visiting (and recording) every live object. With inline types, the objects become coarser, since they now embed inline type contents, which makes GC work easier. On top of that, inline types make the objects denser, so the work for moving the objects gets easier, the allocation pressure goes down, GC frequency goes down, etc. Inline types nicely complement the improvements in concurrent GCs: inline types make less garbage and less processing work for GC, and then concurrent GC do the rest without stopping the application much.