Microsoft Surpasses Java's Dynamic Language Support?

When .NET was first released in 2000/2001, the Java community considered it a "clone" of Java, both language and standard library. Comparing simple code samples surely support this impression. However, MS profited from many years of experience with Java, and managed to solve some issues that Sun only now realizes as problems. The impression that the .NET and the CLR are evolving faster than Java is not lost on the Java community.
Neil Bartlett:

I think it’s clear that Microsoft are innovating faster on the CLR. For example: LINQ is an extremely powerful new feature (based on Haskell monads, I might add); Generics were supported earlier and better in C# than in Java (both flavours were inspired by Haskell’s polymorphic type classes… hmm!); the CLR has better support for multiple languages than the JVM; and now it has the DLR,  which is probably two years ahead of the JVM being able to offer anything comparable.

Other  examples of this are modularization and versioning, which.NET solved by choosing the assembly, a collection of classes, as the basic deployment unit. Assemblies are equipped with metadata such as version information, unlike Java's Jar file which lack  versioning metadata. This is troublesome for increasingly large applications, which load many libraries. OSGi now provides a solution for this, Sun is busy adding something similar to Java 7.

The Java language keeps on catching up with C#, adding features such as Generics and features such as AutoBoxing,  Enumerated types or Annotations. C# now has anonymous expression support, which forms the underpinning of the LINQ technology. LINQ can be thought of a statically typed query language for many different types of data sources, such as XML, relational databases, but also arbitrary object graphs. The Java space, meanwhile, debates language minutiae such as language support for properties and which of four types of anonymous function to include in the language.   

With the release of the DLR, Microsoft has moved ahead again, this time in the area of support for  dynamic or scripting languages on the CLR. The Java space does not have a comparable effort going on right now. Miguel de Icaza, who started the Mono project, a clean room implementation of .NET, summarizes the  features of the DLR:

• A shared type system for dynamic languages.
   
• A shared AST that can be used by language developers to create new dynamic languages.
   
• Helper/utility routines for compiler developers.
   
• A common hosting interface, to embed the general purpose scripting language interface into your program, and allowing developers to extend applications with one or more dynamic languages.
   
• Console support, they even have a simple console interface for doing interactive programming.

The shared type system is an important factor that will allow dynamic languages to interact and exchange objects. Jim Hugunin explains the rationale behind this and shows the situation in Java and how the DLR uses a different approach. Jim Hugunin should know the difference, after all, he developed the  Jython project, a Python version implemented on the JVM, only to later move to Microsoft and implement IronPython, a Python version on the CLR. Jim Hugunin explains one of the issues with this:

The wrapper approach can also have deeper problems. One challenge is just to figure out what object to pass.
For example, if Python has a PyString and it calls a C# function that expects an Object, should it pass the
PyString or should it unwrap it into a String? These kinds of subtle type issues never have a good answer.
Even worse are the nasty problems that can be caused by loss of object identity when objects are silently
wrapped and unwrapped behind the programmers back.  

These kinds of problems are definitely available in the Java space, for instance in the way JRuby 1.0 will handle passing Strings between Java and Ruby code:

• Java strings passing into Ruby code will be encoded as UTF-8, with the implication that you should expect to be working with UTF-8 byte[] in the receiving code
 
• Ruby strings passing out of Ruby into Java libraries will be assumed to be UTF-8, and the resulting string on the Java side of the call will reflect that assumption
 

The Java space doesn't really any of the mentioned items, except for the hosting interface, which was added in Java 6, under the name of JSR 223. This is basically just framework to add new language runtimes and initialize and access them in a standardized way.  

Jim Hugunin continues with a detailed explanation of how dynamic method dispatch is handled, which makes use of extension methods and other existing CLR systems.  The only comparable initiative is JSR 292, which among other things wants to add a new bytecode invokedynamic .This effort was started by Gilad Bracha, who soon after the creation of the JSR, left Sun, and is now not convinced that this project will bring any short term solutions:


JSR 292 is an initiative I started to address these problems. I hope it will continue in my absence, but it will be some years until it comes to fruition (if ever). It is frankly much harder to add support for these features to a JVM then to stabilize Strongtalk.

Note: Strongtalk is a Smalltalk implementation, whose VM is the base for the Hotspot technology, that has been shipped with Sun's JVM for a long time now.

Danny Coward, Specification Lead on JSR 292, feels more confident about performance gains:

Creators of dynamic language engines are in the business of taking, say, Ruby code and turning that into Java bytecode. When today's JRuby engine tries to convert that method call into bytecode, it has to create a synthetic interface to represent the return type. That isn't an interface a developer creates, but is one purely created by the JRuby engine [so] that it can take that method invocation and [turn that into] the bytecode. And that was just the return type—the same holds true for method parameters and exceptions.

JSR 292 removes the need for that synthetic interface. Today, dynamic language interpreters must output the methodinvoke bytecode, even when interpreting, say, a piece of Ruby code. Tomorrow, with JSR 292, they will be using the invokedynamic version. It will streamline the engine implementations because a lot of today's engines are worrying about creating new synthetic types and doing lots of book-keeping: when a method is called in seven or eight different places, they have to re-use that synthetic type all over the place.

It is important to note, that these changes will go into the JVM Specification, which means that they'll be hard coded and will not be easy to update in the future. A library based approach has the advantage that if a better way of handling these systems is found, it can be used immediately. The JVM based approach will remain the same for a long time, because JVMs tend to have a long shelf time (as a reference: Java 1.3 is still in use in companies).  It also remains to be seen whether the JVM can actually make use of this bytecode and really improve the speed of dynamic method invocations.

Another issue is the official support and backing of JVM based languages. Currently, JRuby has two of its developers on the Sun payroll. One of them, Charles O. Nutter, has started to reach out to communities such as Jython and Groovy, and it remains to be seen if these efforts will start.  Considering that MS has teams closely cooperating on IronPython, IronRuby, and the JavaScript and dynamic VB support, it has a certain edge here.After all, the DLR is a product of different teams sharing their experiences and factoring them out into a common library and knowledge base, whereas the JVM based language teams often have to relearn important lessons.

For instance, JRuby features a Just In Time (JIT) compiler, that will turn Ruby code into Java bytecode at runtime. The problem: in the current version, this code breaks the usual  set_trace_func based debuggers (which use a callback approach to implement debugger functionality), by not calling this callback. This means, that the debugging story of JRuby is impacted by this. This same issue must be handled and solved for every language, so sharing at least some of the experience and maybe code, will save time and work.

To add some perspective: Microsoft's DLR must still prove itself, at the moment it is available online together with IronPython. IronRuby hasn't been released yet, and it's real world speed and general interoperability remain to be seen. Java still has the benefit of being viewed as more open and is available on more platforms than .NET, although Miguel de Icaza seems convinced that Mono can ship  support for Silverlight by the end of the year.

May 10th update: Added a "?" to the end of the title to reflect the lack of consensus on this issue that came out in the discussion thread - Floyd.