AdoptOpenJDK has publicly released a suite of tests designed to ensure functional correctness, performance, and overall efficacy of each AdoptOpenJDK release.
The AdoptOpenJDK Quality Assurance (AQA) aims to make the release more in line with expectations of enterprise customers, providing a consistent location for organizations looking to remain up to date on Java without cost. While Java is free, the Oracle JRE became a commercial product requiring a commercial license since Java 11. Users seeking to avoid this clearly-communicated and frequently-publicized change have sought alternative JREs, such as AdoptOpenJDK or the public GPL builds -- both of which are available at no cost.
While AdoptOpenJDK provides a full and complete build of the OpenJDK source code, this new quality assurance program provides more comprehensive validation beyond the Java Compatibility Kit (JCK). In addition to basic conformance testing, the tests include more details on performance, security, and scalability. For example in April, the Java version provided by Docker was functionally compatible but did not contain a series of security patches that left it vulnerable to several exploits.
The first iteration of AQA provides over 30,000 tests, most of which are small enough to run on a developer laptop. Some "special" tests are excluded as they need special hardware, such as multi-byte character sets. The compactness of tests is intended to encourage those working on OpenJDK to run tests before commit, rather than requiring those tests to be run by a continuous integration server. The special tests aim to detect issues that require unique configuration or hardware, such as the 2008 character-encoding issue in Turkey that resulted in a murder/suicide.
The AQA also features a series of performance tests that stress systems, to ensure that each release retains good performance. Unlike SpecJBB and Renaissance, which aim to compare performance between different Java implementations, the AQA tests are self-referential and are designed to prevent regressions in which a release could become slower or consume additional resources. The AQA Jenkins test suite provides status of successful runs, where successful runs of this test take 15 minutes or under. Tests include custom benchmarks as well as incorporating performance tests from popular open source projects like Kafka and Jerry. This expands coverage to prevent issues where the JDK may run its own test faster but inadvertently cause a slowdown in other projects.
While most Java users will not interact directly with the AQA, the impact will be felt more downstream by those that download an AdoptOpenJDK binary.