Interview with Sven Efftinge on Xtend and Xtext

1. Hi, I’m here with Sven Efftinge at EclipseCon 2013. Sven is the project lead for Xtext and Xtend, working at itemis. You’ve obviously been involved with Eclipse modeling for quite a long time, can you give us a little bit of background about how Xtext came to be created?

Yes. So I started my diploma thesis by rewriting a code generator framework, which was known as “Open Architecture Ware” – I don’t know if anybody knows that anymore. During that phase I built languages to do code generation and I had to build Eclipse plugins for that and I figured out I had to do all these plugin stuff, the Eclipse editor stuff over and over again for the different languages. And one day I thought about how could I simplify this by rewriting all that information from the grammer and I did a prototype for Xtext and then it got quite well received and we founded a project at Eclipse. I founded a nice company called itemis, who is paying me for developing this kind of stuff, together with six smart people and my office in Kiel and that’s how it came.

2. Under the covers I think Xtext’s use Antlr is a parser, which is, of course, another way of doing these things. What were the key benefits of using Xtext instead of writing down the Antlr level for doing parsing?

Ok, so the main difference is that Xtext is not a parser generator framework, I mean it uses Antlr to generate a parser from the grammar but it also generates you a fully fleshed Eclipse editor and IDE integration including workspace, index and all that stuff, also you get quite nice hooks for doing scoping and validation, compiler checks and all that stuff. So there’s a lot more to Xtext than just the parser generator. Also from that grammar it un-parses the generator which does not rely on Antlr but it has been written by us.

3. Antlr has different ways of parsing a document for Xtext or for Antlr?

Different ways?

Alex: You were saying the un-parser.

It’s in un-parser. It’s a serializer. You can parse a text in-memory model and then you can do modification on that model and write it back to text again.

Alex: Ok, so you can parse a document bi-directionally

Yes, the grammar is bi-directional.

4. And then, of course, Antlr is only a one way system, how about that. So Xtext is the parsing language; you then created something called Xbase, and then more recently on top of that, Xtend. How did those come about and what are the differences between Xbase and Xtend?

Ok, with Xtext a lot of people build these structural domain specific languages like it’s a great replacement for Xml scenarios, like you get better editors and so on, but you have this problem that at some point you want to have expressions in your DSL, if you take Ant or Maven or so, you want to have some kind of expressions to compute values there, and implementing an expression, especially a statically-typed one, is not very simple to do and also they are mostly the same, especially when you are in the Java environment, there’s no need to have different expression languages over and over again, because they are hard to learn and so on. So what we’ve built with Xbase is a library which you can reuse in your simple DSL. So you concentrate on your structured stuff, like services or entities or whatever you have and when you say I want to have methods in my entities then you just say the implementation is an expression, a Xbase expression, and Xtext does all the type analysis and the content assist and the IDE support and everything for you, you just say “Well this is an expression” and everything works.

5. If I want to create my own mini scripting language DSL, if base it on Xbase, I get a leg up in that most of the parsing and editor generation and static analysis is already done for me, and I just need to add like the keywords and the various operations and the language support?

Yes, in fact we have a very need abstraction there, that is you have your domain specific language AST and you just explain how these maps to a Java model and you put your expression into the scope of that Java model and then the Xtext framework can figure out the scoping rules and everything and links everything for you and it just works then.

6. Xtend is an example of those DSLs that are based on top of Xbase, is it really an exemplar of the Xbase library in that sense, as well as a standalone scripting language?

Xtend is a general purpose language which is built on top of this library, so in that sense it’s an example, it shows that you can really do production ready stuff with that, with very advanced tool support, but of course that’s not the purpose of the language, to be an example. The purpose of the language is really to have a lightweight tool for the typical Java limitations. So in typical Java projects, you very early hit some limitations where you call out to other solutions like XML template languages or what so ever, so if you are a Java developer, you are a polyglot developer because you have so many different formats involved in your Java project. And I think these really have integration problems, so what we usually do as Java developers is we is we pile tools on top of the problem, if you have a problem in Java, well, write a plugin for Eclipse or whatever or a tool or code generators and all that stuff. So Xtext is a great solution for this code generator stuff, but sometimes it might be a bit heavy-weight, because you have to deploy this extra language and keep it up in sync with your project and so on, so with Xtend you can solve many of these problems with more generically, more in a library approach, so we have both, depending on what you need.

7. There’s a lot of interesting JVM based languages, particularly JVM based languages which throw the cup down the amount of craft that Java has inside it, so things like Scala and Groovy are both popular alternatives to Java in the JVM space, how does Xtend fit with those existing implementations or and what are the keys differences and approaches between the two?

With Xtend we’ve focused on interoperability with Java a lot, so it’s not only interoperable in the sense that you can call out to Java but it should be seamless, so the idea is to have a Java project as soon as you think a certain problem is not very well supported in Java, you could just instead write an Xtend class and it will just match into this, so for instance, the type system which is used by Xtend is exactly the same as in Java, so there’s no difference, no gap. Some user said it like: you’re not thinking in two worlds; so there’s really no integration issue. So that was very important and that’s one important difference. Other than that, we’ve focused on very good tool support from the beginning, because that’s important to Java developers, they are used to that and it’s similar to Scala in the sense that it gives a lot of power to the developer and responsibility so you can design very neat abstractions with it and it’s very different from Scala in the sense that it’s much simpler and because it just lives within the Java ecosystem, no extra library and so on.

8. You mentioned a little bit about compatibility being paramount in development of Xtend, what’s the compatibility story for compiler classes between different major and minor version releases of Xbase and Xtend and so forth?

We are an Eclipse project, so we take that seriously, we have certain ways of flagging API as provisional and we do so and if it’s not flagged as provisional, it will stay as it is and it will be compatible in the next release.

9. So if you’ve got a Xtend 1 project that we’ve had for some time, it will still run today on Xtend 2.4, without any changes provided that you stick with the non provisional APIs of course?

Yes, but there are some exceptions, they are more things that sometimes we forgot about a certain compiler check, so there is something like a “feature” maybe which was actually a bug and there are two or three occasions where we decided we really want to fix this. So on the source level, you might get an error message which says: Well, you should do it like this; instead on the compiler level it will be compatible.

10. I mean, if people have put in a Xtend 1.0 libraries into, something like Maven Central and you were consuming its inner project that’s built against 2.4 that would be a perfectly fine from a runtime perspective?

For Xtend 2.4 you would need the latest library but the other way around that would work.

11. If I’ve built something as a library myself that depended on Xtend 1.0, I could still use that library today and against a Xtend 2.4 project without worrying about whether it’s compatible?

Yes, the library is very thin anyway and we don’t do breaking changes there.

12. How large is the library, what does it add to the deployment pipe line?

So the Xtend library is a very thin façade to Google Guava and the JDK, it basically only puts extension methods so the very useful stuff from Guava is put as an extension on the cool stuff from JDK, so to say. And this thin library is, I think 49k or so, just a couple of classes and then there is Guava, I’m not sure how big it is, but it’s not too much.

13. There’s no huge libraries that you need to have. Does that make it more applicable for mobile environments then?

Yes, for sure. With Android you have this limitation that you want to have an application with a very small footprint and with other solutions. I mean most other JVM languages don’t even work well on Android, but if they work they have these big extra libraries, so they usually can’t use the JDK very well because then you can’t use the extra features.

14. Now we talked a little bit about Scala. One of Scala’s key strengths or key selling points is the adoption of, if you like, functional-based programming and high order types for some extent as well as type influence. How does Xtend compare against that sort of programming mind certain model, is it something you can use or something you can’t use?

We have fully embraced this kind of programming style, so one key ingredient is that you have a very neat syntax for anonymous functions Lambdas. Xtend has the most concise Lambda expression out there and the other thing is that everything is an expression, there are no statements, so you can nest if and for as well, like in Scala. Also the type inference is very neat in Xtend, so it goes very far and it does so with the Java type system. So a lot of people complain about generics and to some extent for good reasons, but with Xtend you don’t have to care so much because you almost never have to write this signatures down.

15. And so you get the kind of conciseness of something like Groovy where you don’t have to specify the types of variables but you get the same performance characteristics of Java and the fact that everything is statically typed like Scala?

Yes, exactly, and even more important, you get good tool support and integration with the Java world, so if you do open call hierarchies or open type hierarchies and rename refactoring, it works across Xtend and Java, just as you expected.

16. Does the Xtend IDE replace the JDT like Groovy does or use Equinox Weaving to patch the JDT on the fly like Scala does?

No, we are good participants; we have to being an Eclipse project we can’t use Aspect/J stuff in order to hook into other libraries. We use the possibilities which are available in order to integrate.

17. Ok. So it’s possible to have like you do in Java, one project working on Xtend 2.4 and another project working on Xtend 2.0 and have both of those available in the work space at the same time without worrying about differences between them?

Yes. You can only install one version of Xtend as in the plugin, right? But other than that, you can do that and also you don’t have a conflict with JDT, so we don’t change what JDT does.

18. We were talking about good tool support and type integration with Eclipse. There’re still a lot of people who quite like other developer environments whether that’s NetBeans or IntelliJ. Although there isn’t any support at the moment for Xtend in those environments, do you think that at some point in the future we’ll see developer environments in other IDEs as well?

I hope so. We don’t have concrete plans, we’ve built some prototypes also for bringing Xtext to other platforms, that’s also very interesting, to be able to generate editors for other platforms. We had a short talk in the Orion session today where a colleague of mine is showing how a web editor is generated from a Xtext grammer, so we were working at this kind of stuff but what the Eclipse IDE does for Xtend today, that’s quite a lot and I would love to see other IDEs that have this kind of support for Xtend of course, but yes, it would be cool to have contribution in that direction somehow.

19. We’ve recently seen Xtend release 2.4 and it has added some new features, one of them being active annotations, I wonder if you could explain a little bit more about what this adds to the language and why it might be useful?

Yes, that’s a very neat cool feature. Think of it as annotation processing done right, more or less, or Lisp macros in a static world. It’s basically a way to enhance a compilation step from Xtend to Java source code. So what you do is you declare an annotation and then you implement a compiler hook, where you can get in the annotated element and you can modify it. So you get something like Java lang class object more or less, but it’s mutable, so you can change the stuff and then it’s generated down to Java code. And the good thing is that you don’t lose the static typing, so the IDE is fully aware of your changes, so for instance if you have an add property for a field you will get content assist for the gathers and setters which have been generated in the background.

20. You talk about the classes being translated into Java source, is that the same for all the Xtend compilation and ultimately you just hand over to the Java compiler?

It’s for Xtend and Xtend it’s translated into Java source code and Xtext is not Java specific, so you can build almost any kind of language and translate it into any kind of stuff. We have a lot of customers in the automotive industry for instance, where we generate C embedded stuff and so on. But for Xtend, the compilation’s target is always Java source code.

21. Ok, so you could use Xtext as something to create Java source files quickly and then just put those through an existing build process if you wanted to. Is there any ways of hooking Xtend into an existing build process where it would generate those sources on the fly?

Yes, sure. So you can use a Maven plugin, also the compiler is not Eclipse dependent at all, so you can just call it from command line if you want to, and we provide a Maven plugin to make it easy.

22. What about other build systems there Ant for example and maybe Gradle?

There’s a Gradle plugin, someone from the community built that and Ant should be quite easy but I’m not sure if anybody did that so far.

23. Finally, what’s the future road map looking for Xtend and Xtext?

The future road map is to reduce turnarounds even more so that’s an interesting part and also focus on this compilation thing you can do, so with active annotation that’s one idea but we want to do something similar to Scala macros as well, because we have this cool infrastructure which is very transparent, you get the Java source code, you can read it and so on, that’s quite powerful and other than that we want to make it more performant and add some really slick IDE features and so on, this kind of stuff.

Alex: Sven Efftinge, thank you very much!

Thank you!