The Long Road to Groovy 3.0 Featuring Their New and Improved Parser

The Apache Foundation has released version 3.0 of Groovy, with new features including: a new parser, package namespace changes, an enhanced Elvis operator, and support for Java syntax such as the do/while loop, array initialization, lambdas, and method references. Groovy 3.0 requires JDK9+ to build applications and the minimum supported version of the Java Runtime Environment (JRE) is JDK8.

Groovy 3.0 introduces a new Groovy parser, codenamed Parrot. Its name was derived from its intent to "parrot" output from the now-heritage Antlr2 parser. Parrot was designed to be more flexible and maintainable, and also supports additional syntax options and language features.

When asked about the differences between the heritage and Parrot parsers in the summer of 2018, Paul King, principal software engineer at Object Computing (OCI), Groovy committer and VP Apache Groovy, told InfoQ:

When we speak about the "Parrot" parser, we are talking about Groovy 3's totally revamped early stages of the compiler. It has been re-engineered to be much more flexible and using recent well-supported technologies. This work puts Groovy into a great position to evolve quickly which means we can incorporate changes from Java for "cut-n-paste" compatibility with Java as well as native Groovy changes.

When we speak of the "Indy" support. This is all about the kind of bytecode produced in the later stages of the compiler. In Groovy versions 2.0 through 2.5, we support producing "classic" Groovy bytecode as well as an "indy" version. The indy version makes use of the "INvoke DYnamic" bytecode instruction introduced in JDK7.

The Parrot parser is enabled by default and the heritage parser will remain deprecated for now. Developers who may have problematic code that won't compile with Parrot may disable it and revert back to the heritage parser by adding a system property, -Dgroovy.antlr4=false, on the command line. The heritage parser will, however, ultimately be removed in Groovy 4.0.

In preparation to support the Java Platform Module System (JPMS), first introduced in JDK9, and to accommodate new Groovy modules to be considered first class modules, several Groovy classes have been moved to a different package namespace. For example:

• groovy.util.XmlParser will move to groovy.xml.XmlParser
• groovy.util.XmlSlurper will move to groovy.xml.XmlSlurper
• groovy.util.GroovyTestCase will move to groovy.test.GroovyTestCase

For backwards compatibility, these classes will be available in both package namespaces, but the heritage namespaces will ultimately be removed in Groovy 4.0.

JDK14 early access (EA), available to the Java community in the summer of 2019, provided a preview of anticipated new features. However, some unexpected behavior was discovered with the combined use of JDK 14 EA, Groovy 2.5.x and Gradle 5.5.x. Executing groovy -version and a Gradle task, such as gradle clean, on the command line would throw an exception. Interestingly, executing gradle -version would work as anticipated.

Gradle utilizes the Groovy-based domain-specific language for project configuration in favor of XML as used by Maven. The underlying issue is that Gradle depends on Groovy 2.5 which reflectively calls a private constructor in java.lang that has been removed in JDK 14.

Stephen Felts, senior software development manager at Oracle, documented:

This is broken in JDK14 b8. This is a bug in groovy-git/src/main/org/codehaus/groovy/vmplugin/v7/Java7.java

It seems to be hacking one of Lookup's constructors. The constructor is private and does not exist in JDK 14 as we've refactored this area as part of the work to fix the issues teleporting with Lookup objects.

There is a try/catch block but it is catching a SecurityException. This type of error starting in JDK9 throws a new JDK9 runtime exception. To write portable code, you can't catch the exact exception but instead use catch (RuntimeException).

This issue was resolved as Groovy 3.0 evolved, but led to breaking changes. Asked about these breaking changes in Apache JIRA, King responded:

It depends on how much individual projects decide to tackle as part of moving to Groovy 3.0.

As just one example, as part of JDK9+ package naming conventions within modules, groovy.util.XmlParser must eventually be renamed to groovy.xml.XmlParser, otherwise there would be a groovy.util package in both the groovy and groovy-xml modules.

For Groovy 3.0, Groovy's "modular jars" aren't strictly conforming to JDK modules and we in fact have both classes available. So, for a project like Gradle or CodeNarc or whatever, if it happens to use XmlParser, it could decide to do very little now and use the old class (but have some work to do when moving to Groovy 4) or can do a little more now to avoid extra work later.

King spoke to InfoQ about the challenges with this latest release and what to expect in Groovy 4.0.

InfoQ: What would you like to share with our readers about Groovy 3.0?

Paul King: Groovy 3 is a big release with about 600 bug fixes, improvements and new features. There are 80+ new GDK methods, improved ways to embed Groovydoc and new AST transforms, but the release is mostly known for its brand new parser.

The new parser is mostly an internal detail which makes the Groovy team's life easier since the new parser is much more flexible. We used the new parser to add several new features to the language like new operators (such as the '===' reference equality operator, as well as the '!in' and '!instanceof' operators and an Elvis assignment operator).

Also supported is safe indexing, for example list?[index] will return null if the list is null. There are also many Java compatibility changes which are now supported including lambda expressions, method references, the var reserved type, do/while loops, some array initialization syntax, interface default methods and even the new '\s' backslash escape sequence from JDK14.

With all these Java compatibility changes, it might look like a big part of Groovy 3 was playing catch up with Java. That's not really what is going on. Groovy often supports more succinct and idiomatic ways of coding certain Java constructs, but also supporting Java syntax greatly reduces the learning curve for developers coming to Groovy from Java.

InfoQ: What were the challenges related to JDK9 as a minimum build requirement in Groovy 3.0 that lead to issues using JDK14, Groovy 2.5.x and Gradle 5.5.x?

King: There were a few related issues:

• JDK9 and above started deprecating and removing so-called internal classes, some of which were used by Groovy (as well as many other libraries); we have moved away from those internal classes.
• JDK9 and above used an approach for emitting illegal access warnings which wasn't dynamic language friendly; we now sometimes call methods via parent or interface method signatures (similar to the approach used by Clojure)
• Groovy relies on other libraries (like ASM) for reading bytecode so we must wait until those libraries support particular JDK versions.
• Groovy has a JDK plugin system to use different code for different JDK versions in just a few places; we made it smarter to lazily load old classes which might not be around in later JDK versions.

This work was targeted for Groovy 3 but some parts were back-ported into Groovy 2.5.x.

InfoQ: Will Gradle be forced to rebase to Groovy 3.0 moving forward as a result?

King: Currently, all the functionality needed by Gradle to work with JDK14 is in Groovy 2.5.x, so there is no immediate pressure for them to move to Groovy 3.

However, moving to Groovy 3 would allow Gradle users to use Groovy 3 features such as lambda expressions in their build scripts. It would also allow Gradle to turn off configuration settings which hush illegal warning messages.

Eventually, there will be JDK versions which are only supported on Groovy 3 and above, so we'd encourage Gradle to start looking at upgrading.

We are pleased with the progress made on porting to Groovy 3 by other libraries in the Groovy ecosystem like Grails, Micronaut and Spock.

InfoQ: What features are planned for Groovy 4.0?

King: Some of the work is largely internal refactoring. Recent versions of Groovy have supported invoke dynamic (Indy) flavored bytecode as well as classic bytecode. Groovy 4 will be all Indy based.

Groovy 3 supports the already mentioned new parser, but also allows you to switch back to the old parser. Groovy 4 will support only the new parser.

On the new feature front, we are looking at numerous features, some of which are still experimental. We are looking at a Groovy equivalent of .NET's language integrated query (LINQ), native support for writing module-info files in Groovy, native support for writing records in Groovy, improved support for platform logging, and a new JavaShell class for working with fragments of Java.

We may also look at native support for Java 12/13 switch expressions (though Groovy's existing switch has more power in numerous ways).

InfoQ: When can the Java and Groovy communities start seeing alpha and beta versions of Groovy 4.0?

King: We are actually ready to release the first alpha of Groovy 4 soon. We know it is far from complete, so it is a matter of deciding when to release and setting appropriate expectations. I would anticipate a release in the next few months.

InfoQ: What else is on the horizon for Groovy?

King: We will continue to look at improving Groovy for scenarios where our users are keen to make further productivity gains.

Some users are using Groovy with GraalVM for performance gains, but it isn't ideally suited to dynamic languages. We'll look at ways to make Groovy work better with GraalVM. We are also seeing an uptake in Groovy by Data Scientists so we'll continue to improve support for them.

Arguably Groovy's biggest contribution as a JVM language is its extensibility. We will use this extensibility to continue improving both the dynamic and static natures of Groovy.

Groovy celebrates a milestone this month as the number of Groovy artifacts downloaded from Maven Central and Bintray surpasses half a billion. Groovy 3.0 can be downloaded from the download page and full details about this latest release can be found in the release notes. Developers can expect improved JPMS support in Groovy 4.0 with alpha versions planned to be released soon.

Resources

• Apache Releases Groovy 2.5 and Preview of Groovy 3.0 by InfoQ (July 12, 2019)