MagLev: Gemstone builds Ruby runtime based on Smalltalk VM

OODB vendor GemStone is working on a Ruby VM called "MagLev". InfoQ talked to MagLev PM Bob Walker to get the details behind the project and Avi Bryant who is also involved with the project. First our conversation with Bob Walker.

InfoQ: What's the relationship between Gemstone/S and MagLev on a VM level?

MagLev and Gemstone/S certainly share a lot of code and capabilities, but they are separate products, including at the VM level. A number of the bytecodes and algorithms used by the MagLev VM are specific to Ruby. However, it does retain the ability to run Smalltalk code.

InfoQ: How many people are working on the project? What's your time plan for now?

There are around 8 people who are working on MagLev at least part time. I can't really discuss timelines right now, but will have something to say on this subject at RailsConf.

InfoQ: What's MagLev written in? Is it 'Turtles all the way down'?

Our goals are similar to those of the Rubinius project: to have all of the methods in the standard library implemented in Ruby, apart from a few performance-sensitive primitives. The VM is written in C.

InfoQ: Do you generate bytecode for the VM or how does the Ruby hit the VM?

Yes, we produce bytecode for the VM, which can optionally produce native code for several architectures.

InfoQ: What do you use for parsing Ruby? (the pure Ruby ruby_parser or something you built in house?)

We have the ability to translate from a number of parsers to an internal representation which we use for compilation. We have not yet decided which parser the final product will ship with.

InfoQ: Do you use anything from the Rubinius project (besides the Ruby spec)?

I've been using the Rubinius bm scripts and tests to collect up performance numbers for the various Ruby implementations. I've also looked at the MSpec implementation, scripts and tests to see what might help us guarantee MagLev's API level compatibility with the evolving Ruby standards. That said, I'm also examining the same bits in several other Ruby implementations.
We do hope to eventually share as much of the standard library code as possible with Rubinius.

InfoQ: What kind of performance are you shooting for for 1.0?

We're much more focused on scale than we are on raw performance. Although we believe our performance will compare very favorably with other implementations, we believe our scalable persistent architecture will be the most interesting differentiator.

InfoQ: What's your threading story for MagLev? Native 1:1 threads, an m:n model or plain userspace threads?

Each individual VM has plain userspace threads. However, multiple VMs can have transactional concurrent access to the same objects, so you end up with something similar to an m:n model.

InfoQ: Considering you'll be at RailsConf, is it safe to say you're shooting for Rails compatibility? Or something else first (like merb)?

That's a really interesting question, and one we're putting a great deal of thought into. We definitely plan for MagLev to be Rails capable/compatible. It's not a simple thing to accomplish, as there are a tremendous number of potential (and actual) gotchas we know we'll run into along the way. The way I see it is we must be able to fully run Ruby before taking on Rails. We're also looking at alternatives, and will likely be able to support anything that is written in Ruby. Prioritizing what to do and in what order depends, in a large way, on the feedback and interest we receive from the Ruby Community.

InfoQ: Are you going to make ActiveRecord work with the OODB or will users want or have to use specific APIs with MagLev?

Making ActiveRecord work with the MagLev DB is a highly desirable goal, although there are portions of the ActiveRecord API that assume object state is stored in an SQL based RDB. That could get sticky. In any event, whatever API we have underneath ActiveRecord would likely be available to those that don't want to use ActiveRecord. It's really too early for me to say much more on this subject, I'm hoping that the community will let us know what type of approach they'd find most beneficial.

InfoQ: Are you planning on MagLev being a drop-in replacement for regular Ruby or will it use some kind of Smalltalk-style Image-like deployment?

It will absolutely be possible for MagLev to be used as a drop-in replacement. For those who prefer an image-based approach, there will be support for that as well, but nobody will be forced to use it.

InfoQ: Will Ruby code be able to persist the process to an image a la Smalltalk?

In short, yes. For example, the GLASS product currently lets you save processes that have encountered errors to the repository, and pull them later into a local VM for debugging. There's no reason we couldn't do the same thing for Ruby.

InfoQ: Are you going to use a similar licensing model as with your GLASS products?

We're looking at a number of different models. I can say (but can't guarantee) that we'll do something similar with MagLev.

InfoQ: Do you have a tools story for MagLev yet? Are you looking into supporting Ruby IDEs as frontend (for debugging GUIs, etc)?

Well, of course we'll have an IRB shell, and a ruby-like command for running ruby code on MagLev. I'd like to see IDE plugins, but can't hazard a guess on when GUI tools might become a reality.
I'd like to mention that the MagLev team plans to reach out to and support the Ruby community, and we hope that community members see some compelling reasons to contribute to MagLev. Our core competency here at GemStone revolves around dynamic languages, scalable virtual machines and native-language object persistence - there are many others out there that are better at tools and UI's, and we'd love to support their MagLev efforts.

InfoQ: Any chance of getting Ruby code on MagLev to interact with Smalltalk code or objects?

The MagLev VM is, in addition to understanding Ruby, perfectly capable of understanding Smalltalk. I'd say the chances of being able to do this are pretty high, but we'd want to see if it is desirable functionality that would be useful to the community before going too far down that road.

Gemstone has been working with Open Source technologies for a while - for instance by providing a solutionf for running the Seaside web framework on Gemstone/S. This has been available as GLASS - GemStone, Linux, Apache, Seaside, and Smalltalk. GLASS is free of charge with databases up to 4 GB in size, with other licenses available as well.

Seaside is the creation of Avi Bryant  one of the promoters of the idea of reusing Smalltalk VMs for Ruby  (also see his article "Turtles all the way down" on the same topic).   In a video interview (soon to be released here on InfoQ), Avi Bryant gives some more information about his work on Seaside and DabbleDB, as well as his involvement with GemStone and MagLev.  Avi says of the project:

I've been lobbying the Smalltalk vendors for years to apply their technology and know-how to Ruby, and it's very exciting to see it finally happen. We're less than three months into the implementation of MagLev, but I'm extremely encouraged by the results so far, and we're going to have some great demos to show at RailsConf. I'm afraid I can't get into specifics until then, so come see our talk to find out more.

In a recent blog post, Avi gave an introduction to the benefits of OODBs like Gemstone's by comparing their solution to the ones built in Rails projects with memcachedb and similar technologies:

What about Gemstone? As it happens, the architecture is exactly the same: there's a single storage engine (called a "stone"), a memory cache on each server (the "shared page cache"), and any number of Smalltalk VM worker processes ("gems"). The gems handle the requests and run the code, and they stash objects in the page cache for speed and in the stone for persistence. The difference is, in Gemstone, these have all been designed from the ground up to work together as quickly and seamlessly as possible.

Note: the blog post goes into much more depth and gives a thorough overview of the solutions and different approaches.

GemStone will present more details about MagLev at RailsConf 2008 in May.