The sponsored solutions track at QConSF2016 features talks from Twitter. Yesterday, Twitter’s JVM team announced their work on developing, releasing and maintaining the company’s own JDK distribution based on a fork of the OpenJDK. It currently supports OSX and Linux, with releases scheduled once a month, or at shorter intervals for emergencies. Twitter’s John Coomes noted similarities between Twitter and how their approach to forking OpenJDK is analogous to what the Oracle-JDK is in terms of a divergent featureset from OpenJDK. When asked about the degree to which Twitter attempts to merge in upstream changes from OpenJDK changes, Coomes noted that it depends on the release feature set and that it’s not necessarily part of their strategy to maintain a tight coupling to the OpenJDK master.
Some of the new feature Coomes noted in Twitter-JDK are extended heap profiling, asynchronous garbage collection (GC) logging, a binary logging framework dubbed Contrail, intermediate generations for G1, additional APIs and command-line arguments, performance improvements and bug fixes. The Twitter-JDK improvements are contributed upstream when possible.
Coomes noted some of the existing problems in the JVM, specifically around long GC pauses and some of the ways in which one can manage long pauses on OpenJDK. One approach is to segregate I/O intensive and latency sensitive jobs. This is tedious though when there are thousands of Twitter services running on mostly Scala and Java. JVM I/O can be turned off during safepoints, but this results in decreased ability to monitor the JVM. Coomes noted deprecation of Python-based services in favor of JVM-based languages, inferring the number of JVM-based services at Twitter is only going to go up. The Twitter-JDK handles some of these challenges by implementing asynchronous GC logging and jvmstat. This results in pauses no longer blocking I/O. Coomes noted the future potential release of Contrail, a utility for asynchronous log GC with a JIT compiler, runtime and stack trace support. Twitter-JVM tuning implements Bayesian optimization, a technique borrowed from machine learning so that the JVM can learn potentially noisy cost functions iteratively, efficiently, and resulting in faster discovery of optimal processing paths.
In terms of deciding what parameters to tune in the Twitter-JDK’s VM, Coomes noted there are too many parameters, around thirty, to choose from for addressing optimization scientifically. The cost function used to measure the VM’s performance is the number of request per second over the wallclock time spent in GC. Twitter ran the new VM in staging with minimal iterative testing of 70 cycles for 30 parameters, with a reported performance improvement of 182 percent. Future work entails stress-testing or red-lining, longer experiment durations, concurrent experimentation with Mesos, and the ability to detect and terminate poorly performing experiments early.
