Obie Fernandez on Agile Ruby DSLs

Software-Engineering Radio, the "Podcast for Professional Software Developers" has an interview with InfoQ's own Obie Fernandez on Domain Specific Languages (DSLs) and how Ruby facilitates writing internal DSLs, summarized here.

A DSL is a language focused on a particular domain (hence the name). A general-purpose language (be it C or Java ) can certainly be used to implement the same code as a DSL, but results in more verbose code and hides a lot of the domain knowledge behind general purpose language constructs (for loops, if conditions, method calls, import statements, etc.)

Generating and maintaining general-purpose code is a problem too: experts have to turn their knowledge and experience into code. Usually those experts (be they salespeople, managers, or gardeners) lack the necessary experience with programming, which means they must use the services of a programmer. Of course, it also means that every change in the code involves many steps and indirections: if a domain expert needs to change something, she has to talk to programmer, the programmer implements change, domain experts checks that the code actually behaves correctly, etc.

One possible solution is development of a custom language that a non-programmer can use to describe solutions to their problems. The custom language removes the clutter of general purpose languages and provides only the types and specialized constructs necessary to tackle the problem domain.

In the interview, Obie likens DSLs to slang or jargon which develops in natural languages. Coffee drinkers around the world will be familiar with this:

Venti half-caf, non-fat, no foam, no whip latte

This is gibberish in a normal conversation, and even most waiters in cafes around the world would probably just serve any type of milk + coffee. Uttered in the right context (i.e. a Starbucks), the phrase will yield the correct beverage with the minimal amount of words and less chance of misunderstandings.

DSLs can be implemented in many ways. One option is to define a grammar and use a parser generator tool (such as ANTLR or YACC) to generate a parser. This will take DSL code and turn it into a data structure (a parse tree) that can then be interpreted. An example would be Make files, which handle defining build processes. Another approach is to avoid the custom parser, and use XML, which mainly exchanges the dependency on a parser generator tool for a dependency on an XML parser and various tools that make XML processing bearable (DOM parsers to get a parse tree-like data structure, XPath for making it easier to extract data, etc). An example for an XML based DSL is Java's Ant, where build processes are defined in an XML format. These are examples of external DSLs.

A different approach is an internal DSL, which avoids parser generators and XML parsers, and defines the DSL using legal constructs of an existing general-purpose language. Clearly, the host general-purpose language needs to have a very flexible syntax to accommodate various concise ways of expressing DSL code.

The ultimate tool for internal DSLs is certainly LISP and LISP-like languages. The reason is the LISP syntax, which can be easily summarized:

• atoms
  nearly any sequence of characters, so foo, :::bar::: or even + are valid atoms
• lists of atoms
  wrapped in parentheses

The LISP macro feature allows developers easy access to take a DSL defined in LISP code and either interpret it or expand it into ordinary LISP code that can then be executed. 

Another language good for internal DSLs is Ruby. This is rather surprising, considering Ruby's rich syntax. It is some features of Ruby that make this possible. Here are some of those reasons with examples from slides of a talk Obie gave on DSLs :

• Function call parameters can be given without parentheses. It might seem insignificant, but it allows this to be legal Ruby code:
  
  

  order = latte venti, half_caf, non_fat, no_foam, no_whip 

  
  In this case, latte and all the identifiers in the call are function calls, with latte returning an object initialized up with the coffee's properties.
  
  
• Class definitions are executed at load time
  
  

   class RuleSet < ActiveRecord::Base   has_many  :commends, :dependend => :delete_all   # ... more...  end 

  
  This is an example of an internal DSL used in Rails ActiveRecord. The has_many call happens when the class is first loaded. The call is used to set up the details of the class and it's behavior. For instance, it can add methods to the class using the define_method call. This permits the user of this DSL to define certain aspects of a class in a very concise, declarative way. As a matter of fact, this has some resemblance to the way LISP macros generate specialized code in that both approaches work at the time the code is loaded.
  
  
• Compact way of writing Blocks
  
  Blocks are self-contained chunks of Ruby code. They can be stored, passed as arguments and executed at will. Other names for this concept are anonymous functions, lambda functions, or closures. Ruby has a very compact way of passing blocks to a method, which makes it particularly easy to implement custom language constructs. Rake, is a build tool similar to Make or Ant. An example:
  
  

  task :default => [:test]  task :test do  ruby "test/unittest.rb" end

  
  This example uses all three concepts mentioned here. task is a function call, but without the parentheses looks more like a declaration. The task calls are executed at load time, setting up the internal data structures. The logic of the "test" task is defined in the block (the code between do and end), and is executed whenever the "test" when Rake determines this task to run.
  

Internal DSLs in Ruby make it very easy to write concise and declarative specifications. And as can be seen from the has_many example, it's also easy to intermingle ordinary, imperative Ruby code with an internal DSL.