Last week saw a flurry of activity in the Scala landscape, with a Yammer mail leak stirring up questions. Typesafe subsequently published a response, graciously noting the feedback and painting a picture where Scala would improve in the future.
Top of the list is the upcoming availability of the Scala IDE for Eclipse, which has been undergoing development for a long time. Originally started by Miles Sabin, the project is now resourced from TypeSafe's employees as well, and currently exists as 2.0.0-RC2. Like other pre-compiled Scala libraries, there are a number of downloads which are hard-wired to a specific Scala version; you can use any one of 2.9.1, 2.8.3-SNAPSHOT or 2.8.1.
Other commercial IDE vendors already support Scala; the recently-released JetBrains IntelliJ IDEA 11 contains Scala support out of the box for both commercial and community editions.
TypeSafe also responded to specific criticisms, such as the learning curve requirements, build times and performance as part of their getting down to work post. They note that Scala is a work in progress, and that whilst performance is important they are keen to avoid any premature optimisations as the language matures. For building, they recommend the Scala Simple Build Tool, or SBT, which operates in a daemon mode to save time between startups of the build chain. And, as part of increasing the community around Scala, have now moved Scala to GitHub.
However, their comments on binary compatibility did not satisfy many. Whilst TypeSafe offers a subscription plan, which guarantees a commercially supported and maintained release of the Scala+TypeSafe stack, they did not consider the greater community and framework angle that has caused the growth of other platforms such as Java, Python, PERL etc. Instead, they offer a closed-source Migration Manager which can statically determine whether there may be any compile-time variation between builds. They note they themselves use this tool to ensure that Scala 2.9.0 was backwardly compatible with 2.9.1. They also claim to aim to provide all frameworks a developer could need, such as common libraries for testing, IO and Software Transactional Memory.
David Pollak, creator of the widely known Lift web framework, isn't so sure that this is the right focus. In Scala's Version Fragility makes the Enterprise argument near impossible, he argues that the fragility of the bytecode is directly harming the growth of the Scala ecosystem outside of the TypeSafe stack.
An attribute of Scala is that the Scala compiler generates fragile byte-code. This means that all the code in an executable (JAR or WAR) must be compiled with the same library and compiler versions. If you're using Scala 2.9.1 in your project, you must compile against a version of Lift that's compiled against Scala 2.9.1 and all the Lift dependencies must also be compiled against that version of Scala. This is a reason that Lift has precious few Scala library dependencies.
Scala's version fragility creates two huge costs for complex enterprise-type systems:
- All dependencies must be compiled against the exact same version of Scala and every referenced Scala library which makes in-house builds when there are multiple teams amazingly complex or time-consuming
- External libraries (like Lift) cannot have multiple layers of dependencies because you need to match the exact version of every library and Scala version all the way down
Unlike others (who maybe have opinions but no direct Scala experience), David is one of the few who can boast five years involvement working with Scala as a language and with the wider community. The operation of the Lift framework project, along with its guiding principles, is a model of excellence in the Scala (or indeed, any) world as how a community should operate.
The problem this impacts is not an issue when you have small development teams, or you are the only team in your organisation using Scala. If you're doing a greenfield migration to Scala for an initial project, the cost of migrating that team to a new release is a matter of less than a day's work. However, throw in two teams and suddenly there's a co-ordination issue that wasn't there before. The problem degrades exponentially:
So, if you're in an organization with more than 2 or 3 development teams that are all generating internal Scala code, you can generally agree on the version of Scala and the various libraries that you're going to use. When a new version of Scala or the libraries are released, you can get a couple of people in a room and choose to upgrade (or not.) But once you get get past 3 teams, the coordination efforts become significant. Trying to get 10 different teams to coordinate versions of Scala and Lift and ScalaZ and Rogue and Dispatch, etc. is going to be a significant cost.
As a library builder, the version fragility issue means that our library is not nearly as good as it could be and we can't service our community the way we want.
David has voiced his concerns before, in InfoQ's Barriers to Scala Adoption article published in October. However, he feels that the issue still isn't getting enough attention and that the closed-source bytecode checker is a plaster on the symptom rather than an underlying fix.
What have I done about the issue? Over the years, I've raised the issue privately and it has not been addressed. I have tried to organize a project to address part of the issue, but haven't had the volunteer uptake for the project.
A year+ back, during the 2.8 development cycle, some of us got together on the "Fresh Scala" project. Basically, the idea was to have continuous integration of the major libraries in Scala-land and weekly milestones so that the community and the ecosystem could give better feedback on new versions of Scala. Due to a lack of volunteer time, we never got the CI system built and here we are a year and a half later with the same Scala version fragility issues.
The Fresh Scala initiative didn't get much traction a year and a half ago, but Josh Suereth of TypeSafe claims responsibility in his Scala Fresh 2.0 post. In it, he announces Scala Fresh 2.0, a reboot of the project with a final destination of GitHub:
What happened to Scala fresh?
People may not realize but Scala Fresh failed because *I* failed Scala Fresh. I had very little time between a more than full time commitment at Google, writing commitments, and kids. This should not be an issue in the future, thanks to Typesafe taking binary compatibility seriously.
I've begun work on what can only be called "Scala Fresh 2.0". That is, a place where community libraries will be built and deployed against the latest version of Scala. This can serve two purposes:
- Ensure that future versions of Scala do not break community libraries
- Ensure that the core libraries of Scala are available for every major version of Scala.
Josh suggests that binary compatibility is the responsibility of library developers instead of the platform:
Binary compatibility is a community effort. I know Typesafe is doing what it can with its resources, and I'm personally tackling as much as I can (probably trying to juggle too many balls). However, if you want to help, please email me!
The last point is worthy of note; it is possible to write Java code which evolves in a backwardly incompatible way as well (removing public methods, changing method signatures etc.). Having libraries evolve where methods are removed or modified will cause backward compatibility issues whatever the language. On the other hand, designing libraries to be backwardly compatible is something that many Java developers automatically do, and tools like PDE's API usage scan can help perform a similar role.
However, where the evolution of Java libraries is somewhat predictable (by virtue of the class implementation being in one file), Josh highlights that it's possible to introduce incompatible changes via traits in Scala, since they are (in effect) an interface+implementation:
trait Foo { def foo = "foo" } trait Foo { // backwardly compatible with the above def foo = "foo" def bar = "bar" } --- trait Bar { def foo = { " foo" } } trait Bar { // not backwardly compatible def foo = { bar + " foo" } def bar = "bar" }
As a result, it's possible to have source code compatible changes (a class which has the Bar trait will still compile in both cases) but clients compiled against the old version of Bar will not have a bar() method, so newer clients will attempt to call a method which has no implementation.
What causes some pain is that Scala's key libraries suffer from this all the time. The Scala collections libraries are tweaked with each release of Scala, and since most APIs use the Scala collections libraries, there is a hard-coded dependency on the version of Scala used to compile (and the core libraries to compile against) in the generated bytecode. Some have started asking whether it is a better idea to ignore Scala as a compiled language, and treat it as a 2.10 development path, and no sign of the backward incompatibility being addressed, it is safe to say that if you want to move forwards with 2.10 it will require another recompile of everything in order to be guaranteed to work. InfoQ asked TypeSafe for comments last week after their getting back to work piece, but no response was received prior to the publication of this piece.