Performance and productivity improvements have gone into recent editions (1.6, 1.7 and 1.8) of Groovy and more are on tap. Recently InfoQ got a chance to catch up with Guillaume Laforge, the project lead for Groovy.
We spoke about the recent history of improvements to the Groovy language, a programming language that builds on top of Java syntax and adds languages features from Python, Ruby and Smalltalk as well as quite a few tricks of its own. We also spoke about what is on the plate for future Groovy development. Guillaume discussed improvements to Groovy for domain specific languages, performance improvements for handling integers, annotation triggered AST transforms, GPars parallel/concurrency library, new built-in language support for JSON, improvements to Grape package management, passing closures to annotations, improvements to closures to support recursive closure calls without fear of stack overflows, and much more. These are features available in Groovy 1.8.
AST transformations streamline writing Groovy code without the boilerplate code, productivity killers, that other languages force you to write. Grape provides Maven/Ivy like package dependency management at the the Groovy script level so scripts can be run standalone and just fetch their dependencies. The ability to pass closures to annotations allows the creation of design by contract style development with Groovy.
Guillaume also discussed Groovy adoption and what is on the roadmap for 1.9 and 2.0 as well as planned support for Java 7 features in Groovy including multi-catch and invoke dynamic. The invoke dynamic support could give Groovy a bump in performance overall. Guillaume mentioned Groovy adoption rates and how well Groovy is doing. Groovy has always allowed you to add types for documentation purposes, but it seems dynamic Groovy is going the Dart route of allowing static compile time checking as well, Guillaume covers this as well.
InfoQ: Can you give us an overview of the latest novelties in Groovy 1.8?
With Groovy 1.8, we worked on several fronts.
First of all, Groovy is well known for its ability to author Domain-Specific Languages, for writing expressive and readable business rules. For that purpose, we introduced a grammar enhancement (called "command chains") that allows developers to create nicer DSLs, by allowing you to make more dots and parentheses optional for chained method calls. With this approach, you can even write plain English sentences in your DSLs!
We also continued our runtime performance improvements, in particular in the area of primitive type calculations. As long as you don't do any monkey patching on numbers or such, and if you specify your variables to be of primitive types, Groovy is able to use the JVM bytecode instructions, like iadd to add two ints together, rather than doing the boxing / unboxing dance and calling methods on Integer and friends. That way, the performance of your primitive calculations becomes much closer to Java than they were with older versions of Groovy.
Besides that, the Groovy distribution now includes the GPars parallel and concurrent library. We've worked on adding a few more utilities further improving the functional flavor of our Groovy closures. We added support for dealing with JSON payloads. And we added several new useful AST (Abstract Syntax Tree) transformations, and some handy control tricks to customize the Groovy snippets you may be integrating in your application (for instance, preventing infinite loops, or such).
For some more details on all these features, don't hesitate to have a look at the release notes of Groovy 1.8 .
InfoQ: What are AST transforms and how are developers using them?
AST Transformations stand for Abstract Syntax Tree transformations. It's somehow a compiler plugin, if you will. The AST is the in-memory representation of your program, made of class, method, property, statement, variable nodes, etc, before it gets compiled to Java bytecode. AST transformations are the ability to hook into the Groovy compilation process, and let you modify the structure of your program, before we generate the final bytecode for the class. With that capability, you are able to add new methods, add checks in your code transparently, generate some other classes, etc. It opens up a lot of interesting possibilities.
There are two kinds of such transforms in Groovy: local and global transformations. Global ones are applied to every compilation unit, whereas local ones are triggered by annotations. For the latter, for example, you could annotate a class with @ToString to invoke the transformation that adds a nice default toString() implementation in your class.
Developers are using such transforms to further simplify the life of their users, by doing some compile-time metaprogramming, instead of the usual dynamic runtime meta-programming that people use with dynamic languages. This has the nice benefit of being a compilation feature, which doesn't add a particular overhead at runtime. And overall, it's generally for making users' code even more expressive and concise.
InfoQ: Can you give us a few examples of the AST transformations that were added to Groovy?
Groovy 1.6 and 1.7 introduced several useful transforms, such as @Delegate for having methods delegated to one of your fields annotated with that annotation, or @Immutable to make your class immutable, as the name implies.
Groovy 1.8 adds several new useful transforms such as the various @Log transforms to inject a logger in your classes and wrap your logging statements with the required "if" checking the log level, @InheritConstructors to add constructors implementing the super constructors and delegating to them (think for instance when you have to extend an Exception class), @WithReadLock / @WithWriteLock to simplify the use of java.util.concurrent's ReentrantReadWriteLock.
There's the combination of @ToString, @EqualsAndHashCode, @TupleConstructor and @Canonical, to add toString(), equals(), hashCode() methods, and a constructor with the various properties you've declared as parameters.
As usual, the objective of all these new transforms is to let developers streamline their code by focusing more on the business code they need to write, than on the nitty-gritty boilerplate they were forced to write with other languages. Productivity of the language is key to successful projects!
InfoQ: How hard it it to write your own AST transforms?
To write AST transforms, you'll need to have some knowledge of the Groovy AST APIs. So it's not totally trivial. But thankfully, Groovy has a few tricks in its bag to help you author your transforms. First of all, with the Groovy console, you are able to see the structure of the Groovy AST so that you can mimick what you see there for the piece of code you want to generate or modify. There's also the AST Builder class which offers 3 different mechanisms for building pieces of AST, from a code fragment, from the code included in a closure, or using a builder API for constructing the AST structure yourself in a less verbose manner than manipulating the AST classes directly. Writing AST transforms is a fair bit of work, but Groovy helps you in this task.
InfoQ: What is GPars? Who uses it? Why was it included with Groovy 1.8?
A few years ago, the Groovy team was interested in the trends of computing which sees our computers becoming more parallel (multiple processors, multiple cores). It was clear back then already that Groovy needed to support this trend, by offering higher level of power and expressiveness to tackle the parallelism and concurrency problems. Java and the JDK offer some primitives like threads, and the various wait / notify / notifyAll methods, and more recently all the nice additions to the java.util.concurrent packages.
Groovy obviously offers some improvements in its core for these aspects, but higher level concepts exists that we wanted to provide to our users. That's how the GPars project, lead by JetBrains' advocate Vaclav Pech, came to be, because we wanted to offer higher level features, such as Actors, Dataflow concurrency, Software Transactional Memory, Agents, and various enhancements to java.util.concurrent's executors, parallel arrays, fork / join, map / filter / reduce, etc.
GPars is a project in its own right, with its own roadmap. Initially, we had thought of merging it inside Groovy directly, but with the community that gathered around the project, and to let it evolve at its own pace, we chose to embed it in the Groovy distribution instead.
In passing, what's also nice with GPars is that it's not just a Groovy parallel / concurrency library, but it also supports a Java API so you can use it from Java as well, without enforcing the usage of Groovy as a language -- although of course the code won't be as compact with Java.
Don't hesitate to have a look at the GPars project for your next concurrent and parallel needs . And in particular look at the detailed GPars user guide.
InfoQ: What is grape and how has it been improved?
Grape, with its @Grab annotation, is a feature that let you define dependencies that your script, or even class, may need. For example, I could add in my script: @Grab("groupId:artifact:version") to an import, and Groovy will download and cache the dependency so that it's available on the classpath when executing your script. It's very handy when you want to share scripts with your co-workers, because you just need to share the script itself, and you don't need to package the dependencies or such.
In Groovy 1.8, the enhancements are around the string value passed to that annotation, and to the @GrabResolver annotation (for specifying a specific repository), to have a compact notation to describe your dependencies and their origin.
InfoQ: What support has been added for reading and producing JSON?
JSON is everywhere nowadays, and more and more REST APIs blossom on the Web only offering JSON payloads. Groovy's always been great at handling XML, both for consuming and producing XML content with its XML builders and parsers, so we wanted to add the same level of support in Groovy 1.8 for JSON. So we introduced a "slurper" that parses JSON content, as well as a streaming and non-streaming "builder" that let you build JSON payloads, in very concise ways, without the need of implementing some special marshalling.
As examples, if you wanted to create a person JSON object, you could write:
def json = new JsonBuilder() json.person { name "Guillaume Laforge" age 34 }Which would generate:
{ "person": { "name": "Guillaume Laforge", "age": 34 } }Of if you parse the commit messages from the Groovy repository mirror on GitHub, you could write this:
import groovy.json.* def endpoint = "https://api.github.com/repos/groovy/groovy-core/commits" def content = endpoint.toURL().text def parser = new JsonSlurper() def commits = parser.parseText(content) // navigate the object graph // and print each commit message commits.commit.message.each { println it }This JSON support is very similar to Groovy's existing XML support, and makes life much easier to handle JSON documents.
InfoQ: You mentioned some improvements to closures, especially for their functional flavor, what can you tell us about these?
Groovy's first-class citizen closures as well as all the methods of the Groovy Development Kit enhancing the various JDK collection classes (each / collect / findAll / etc.) are the basis of the functional capabilities of Groovy. They let you iterate, filter, transform collections and their values, and many more things.
In Groovy 1.8, we added a trampoline() method to closures which allows you to stack up closure invocations serially for your recursive algorithms, rather than exploding the stack with recursive calls. A closure function like factorial would calculate results for up to 40 or more, but fail beyond with a StackOverflowException, whereas a slightly rewritten implementation using the trampoline() call on your closure will let you calculate factorial(1000) and higher without blowing.
Closures are now composable and reverse composable with the left shift << and right shift operators >> respectively, so you can compose the computation of your closures, and get a new closure in return that you can call and pass around.
We've also added a memoize() function which remembers the outcome of previous invocations of your closure for a given set of parameter values. This is particularly handy for calculations that might take a long time, but be careful, as your closures should be side-effect free and always return the same results for the same parameter values, otherwise you'd easily get some surprises in your results! The memoization feature is using an LRU cache, but there are variants of that method which let you have finer control over the cache, by letting you decide how many results you want to cache.
InfoQ: Annotations can now take closures as parameters. Why is it important to pass closures to annotations? What type of annotation could you build that took closures?
Java annotations are somewhat limited in that they accept just a limited set of possible of values: primitives, strings, annotations, classes and arrays of these. But you cannot pass instances of your own classes, or things like that. Groovy goes a bit beyond what annotations allow by encoding closures in the form of closure classes -- so Java is happy to see a Class as parameter. With annotation closure parameters, you annotate fields or methods with things like @Validate({ name.size() > 0 }), if you want to add some validation rules to your methods parameters or field values. This is going to be useful for framework developers as well.
This has been already used by the GContracts library which adds the "Design by Contract" paradigm to Groovy by using annotation closure parameters, for implementing pre- / post-conditions, invariants, and so on. So have a look at GContracts to have a little taste of what you can achieve with that feature.
InfoQ: Which IDE do you use to do Groovy programming?
I'm using Eclipse SpringSource Tool Suite and IntelliJ IDEA.
I've been developing with IntelliJ IDEA for about 8 years or so, so I'm very used to that IDE, and spend more time with it than with Eclipse. But both IDEs really provide awesome support for Groovy and Grails, with all the usual features you'd expect (code browsing, navigation, hovers, refactorings, debugging, running, and more). Both are very good choices if you've developing with Groovy and Java.
I've also heard that NetBeans are in the process of updating their aging Groovy support as well, so NetBeans might get back on track with its competitors here soon.
InfoQ: How is Groovy adoption going? Is it increasing? Does the community seem to be getting bigger?
For Open Source projects like languages or other libraries, it's not easy to track the adoption rates. So you have to look at various indicators, like numbers of persons subscribed to the mailing-lists, download statistics, or even more subjective elements like online or conference polls about favorite alternative languages and such. From those few elements, Groovy is doing very well and is increasing, and seems to be the most popular alternative language on the JVM these days.
For example, if we look back at the download stats from our previous release, in just two months, we had already accumulated close to 400K downloads of the Groovy distribution at Codehaus -- the foundation that hosts the Groovy project. And that number is not even accounting for the Maven Central downloads or the other projects using Groovy like Grails, Gradle, Griffon, etc! So a pretty pessimistic estimate would lead me to think we have at least half a million Groovy users out there! It could possibly be more than twice that number but it's very hard to tell exactly.
InfoQ: What's going to be the next major version of Groovy, and what will it provide?
We updated our roadmap recently to revise our version number scheme a little. The next version will be Groovy 2.0, instead of 1.9 as initially envisioned, since we have added some very interesting features and believe it deserved a full version bump. We have some other exciting changes in store so you might see the version incrementing a little faster than it has previously. We won't go as fast as Google Chrome or Mozilla Firefox in terms of version number increase, but we'll continue on our trend of releasing a major version a year or so. The expected release of Groovy 2.0 should be at the end of the first quarter of 2012.
So what are we working on in Groovy 2.0? Well, three major features: JDK 7 support with invoke dynamic support and project coin enhancements, static type checking, and we're investigating adding static compilation as well.
First of all, for those who will have upgraded to JDK 7 and its invoke dynamic bytecode instruction and APIs, you'll be able to get the performance boosts for all the dynamic aspects of Groovy. The exact gain is still unsure at this stage, but we're confident that this support should continue increasing the performance of the Groovy runtime. Ultimately, for later versions of Groovy that would be based on JDK 7+, we would even be able to get rid of some optimizations we've built over the course of time to make Groovy faster, thus making Groovy a bit slimmer by relying on JDK's own classes and JIT optimizations. For those who are still on JDK 5 and 6, we're looking at the invoke dynamic backport, or into different code path in our code base to continue supporting those older versions of the JDK, so that our users don't have to upgrade to JDK 7 as a requirement for using a newer version of Groovy.
You will also be able to use the Project Coin enhancements of Java 7, such as the multicatch, the binary literals, or underscore in literals -- others like string in switch didn't even need to be implemented, since Groovy's switch accept much more than primitives, strings, etc. What's interesting here, by the way, is that those language enhancements will also be available for older JDKs, and you won't have to upgrade to JDK 7 to get your multicatch clauses, for example.
Next, static type checking. Throughout the years, we've noticed that a good share of our users are integrating Groovy in their applications as a Java scripting language, Groovy being an almost superset of Java. Those users would not use all the dynamic features of the language, and would just code in Groovy the same kind of things they'd code in Java, manipulating their internal APIs, etc. A common complaint of that user base is that Groovy would complain only at runtime that some typos in method names or variables were made, rather than catching such errors at compilation time. So Groovy 2.0 introduces a @TypeChecked annotation which will let you annotate a method or a class for which you want static type checking to happen. It does some clever type inference, checks methods and variable names, verifies correct assignments, and so on.
With all the cleverness of that static type checking mode, the Groovy compiler knows a lot more about your code than ever before, and it opens up an interesting possibility. For those parts of your code base where you're not doing advanced metaprogramming tricks, and that are correctly type checked, we are in a position to generate the same kind of bytecode as Java, bypassing the Meta-Object Protocol of Groovy. The MOP is what makes Groovy dynamic, but in those parts where you don't need all the dynamicity of Groovy, but you want performance as fast as Java, you could use static compilation! And that's what we're investigating in Groovy 2.0. So we're actively discussing those two aspects, static type checking and static compilation, within the Groovy community these days. So don't hesitate to join the discussion there if you want to give your input.