A domain specific language is a language-oriented tool designed to solve a specific set of programming tasks. Common traits are very close to the problem domain within which the language is intended to work, and operating at a very high level of abstraction. In his article about DSLs, Martin Fowler classifies DSLs as either external or internal (see link). An external DSL is a programming language that is either compiledor interpreted. An internal DSL is built within a general-purpose programming language. Effectively, an internal DSL is a very high-level API to a general-purpose programming language. This article is a story about how implementing an internal DSL for Ruby contributed to a PLANET ARGON development project.
The Problem
The goal of one of my recent PLANET ARGON development projects was to build a one-off content management system with Ruby on Rails that supports 18 languages and manages roughly one thousand image files. Many of these files are pieces of professional photography, each one weighing in at well over 1MB. Others are small, detailed pieces of line art bearing various national flags. The only thing these files had in common was that they were not ready to be used in production directly.
Our application has an Image model. It uses a file upload plugin that we wrote in-house to persist image data. We had begun managing the image transformations with after_save hooks that used the RMagick image manipulation API inside of our model. This would have worked well, but our requirements were continuously being adjusted. Some images required 4 variations to be produced. Others required two. Some of them had to be uploaded to a third party bandwidth provider. Sometimes we got the dimensions wrong. At one point, we had written a batch script to upload these source images so the after_save hooks could reprocess the images. Soon enough,the edge cases came and our callbacks became engorged with stacks of conditional statements. As you may have guessed, this got old fast. We needed a tool that would help us meet our client’s needs in a sustainable manner.
The Solution
One day, our client posted another reference image and a set of instructions for how they created the image in PhotoShop, to our Basecamp. I had had enough.Thinking about a more elegant solution to solve the problem, I remembered when Jason Watkins had told me about his experience with a class of tools, called asset compilers, when he worked in the video game industry. An asset compiler is a build system which transforms a set of media files into the media files needed for the final build of a video game. I had recently been automating a lot of repetitive tasks with Rake (Rake is a build system implemented in pure Ruby), so creating an asset compiler with Rake was very attractive. If the duty of creating the production images could be delegated to an external tool, our application would not have to change whenever our client’s requirements changed. To sweeten the deal, this tool would allow us to rebuild the production images with a single command when requirements changed.
I had begun by using Jim Weirich's (the author of Rake) RDoc task library which is distributed with Rake as a template. One usage pattern which Jim has established for Rake is a class which you instantiate with the appropriate options to describe the set of Rake tasks needed to automate your build. For example, here is an example of how to use the RDoc task library in your Rakefile to build documentation for a Rails application, the plugins it uses and the Rails framework itself.
Rake::RDocTask.new('my_docs') do |rdoc|
# configuration
rdoc.rdoc_dir = 'doc/my_docs'
rdoc.main = "doc/README_FOR_APP"
rdoc.title = 'Comprehensive Documentation'
# app documentation
rdoc.rdoc_files.include('doc/README_FOR_APP')
rdoc.rdoc_files.include('app/**/*.rb')
rdoc.rdoc_files.include('lib/**/*.rb')
# plugins documentation
rdoc.rdoc_files.include('vendor/plugins/*/lib/**/*.rb')
# framework documentation
# ...snip...
end
Instantiating a Rake::RDocTask object within your Rakefile as per this example will create Rake tasks to build, delete and rebuild your documentation. The rdoc block parameter yielded by Rake::RDocTask.new is actually the Rake::RDocTask object which is being instantiated, and the options which is being instantiated,and rdoc’s attributes are defined in Rake::RDocTask by attr_accessors.
I began by extracting the code to build the production images from our Image model into a Rake task library. After creating a working task library which operated similarly to the RDoc task library, my code looked something like this:
ImageTask.new :bronze_thumbnail do |t|
t.src_files = image_src 'images/*.jpg'
t.build_path = build 'greyscale_thumbnail'
t.remote_dirs << REMOTE_DIR[:greyscale_thumbnail]
t.transformation do |img|
# apply bronze image effect
img = img.quantize 256, Magick::GRAYColorspace
img = img.colorize 0.25, 0.25, 0.25, '#706000'
# crop to thumbnail size
...snip...
end
end
And that code defined the following Rake tasks, as described by running rake -T:
rake assets:bronze_thumbnail:build # Build the bronze_thumbnail files
rake assets:bronze_thumbnail:clobber # Remove bronze_thumbnail files
rake assets:bronze_thumbnail:rebuild # Force a rebuild of the bronze_thumbnail files
rake assets:build # Build all assets
rake assets:clobber # Clobber all assets
rake assets:rebuild # Rebuild all assets
I realized how flexible my tool would be if the code describing these image transformations closely resembled the PhotoShop instructions given to us by our client – an internal DSL that defines Rake tasks for image transformations.
I created some unit tests for my task library to ensure tasks were being created properly and begun some test-driven refactoring. Because I was creating an internal DSL, which in my case is a Ruby API, the creation of the DSL could be purely test driven. I began by replacing the example code in my unit test which instantiates the task library’s classes with example code for my unimplemented DSL and revised it several times until I was satisfied.
define_image_transformation 'thumbnailize' do
crop_to '62x62', :north
end
define_image_transformation 'bronze' do
greyscale
lighten
# r g b tint
tint 0.25, 0.25, 0.25, '#706000'
end
image_task 'bronze_thumbnail' do
from images 'images/*.jpg'
to build 'greyscale_thumbnail'
remote_dirs << REMOTE_DIR[:greyscale_thumbnail]
transformation do
bronze
thumbnailize
end
end
I ran my unit tests and watched a raft of error messages float by. This was good. I now knew what I wanted, and what steps were necessary to achieve my goal. I then renamed a couple of methods (src became from, for example) and created some helper methods to help instantiate the task library class. I also wrapped the lower-level RMagick code with a a library that used designer-friendly method names and created define_image_transformation which defines high-level application-specific image transformations by aggregating those RMagick wrappers. By making the distinction between what a graphics program like PhotoShop can do and what our client wants to do, I was able to ensure my tool would be fit for extraction and reuse across future projects – the high-level image transformations lived in my app’s lib/tasks directory, but the lower-level wrappers lived in vendor/asset_compiler alongside thetask library.
The Outcome
Currently, our lib/tasks/assets.rake file contains 12 calls to image_task, including the example shown above. Although the language may not be suitable for lay programmers (according to Fowler’s article,lay programmers are not professional programmers, but they can make use of simple programming tools to automate business tasks), it is in sync with our client’s language and neatly models their ideas and goals.
Because Ruby DSLs are Ruby code, I strongly recommend following the test-driven process which I described in this article: code an example of the Ruby internal DSL you wish to use, write test cases for the desired outcome of that example and code until all tests are passing. Also consider the scope of your DSL in the beginning. Because I emphasized a separation between the functionality of an image transformation tool and the needs of our application, building a powerful, reusable and extensible framework around our DSL became natural. I feel that this was the right decision because I know I am going to use this library again for other projects very soon. If your needs are more particular you may not need multiple layers of abstraction. If you intend to reuse your internal DSL, ensure that no application-specific business logic sneaks in, but provide a mechanism for capturing that business logic.
Finally, there is no canonical text on how you should create an internal DSL with any language. The consensus between all articles I have read on the topic is that you must learn by doing. Pay attention to how clients and domain experts communicate their requirements and focus on capturing those requirements with executable code. Even if you can’t transform your domain expert into a lay programmer, they should be able to see their requirements clearly during a code review or a pair-programming session, and your goal should be to capture their requirements as quickly as you receive them.
