GitHub has just made available Scientist 1.0, a Ruby library that will help developers refactor or rewrite their code with confidence, writes GitHub engineer Jesse Toth. Scientist was used at GitHub over the last few years for a number of projects.
Scientist, explains Toth, is particularly useful when refactoring critical code, that is when it is extremely important to reach confidence in the correctness of the new implementation. Particularly, Scientist aims to provide a better guarantee than when using the BranchByAbstraction architectural pattern, which, though very useful in its own, merely ensures that the new component will be used wherever the old one was. Furthermore, it tries to circumvent the limits of testing, which seldom can go so far as to cover all possible cases or input data combinations.
The basic idea brought about by Scientist is that of a controlled experiment whereby both the old path and the new path are stressed, then their outcomes compared, and any mismatches and/or exceptions logged. The old path ensure that the system will continue working correctly during the whole trial period while the new path correctness is assessed.
An experiment is a lightweight abstraction that provides two behaviours, a use behaviour meant to exercise the old path, and a try behaviour meant to exercise the new path.
experiment = Scientist::Default.new "my-experiment"
experiment.use { <call the old code here, the control> }
experiment.try { <call the new code here, the candidate> }
experiment.run
#...
def publish(result)
#...
The run method will always return what the use block returns. The publish method is called at the end of the experiment so collected data can be published. Besides comparing the result of try and use blocks, Scientist will also randomize the order of their execution to avoid any possible correlations, measure the duration of both executions, manage exceptions, and publish collected results.
Scientist provides a number of ways to customize its default behaviour, e.g. by defining a specific compare method to override the default == operator used to compare control and candidate outcomes, specifying a context that will be available when publishing the data, controlling setup tasks, enabling or disabling experiments execution, etc. More advanced control options allow to ignore results, run more than one try block, or run only try blocks to cover special use cases.
InfoQ has spoken with Jesse Toth, principal engineer at GitHub.
Could you describe how Scientist was born?
Scientist was born when a former coworker, Rick Bradley, was trying to refactor a very complex API endpoint that returned long lists of repositories. He wasn’t confident that there was enough test coverage on the code he was changing and wanted a way to test with real datasets. He hacked something quick into the code to call the refactored method and record data to our metrics stack any time the refactored method behaved differently than the original. This worked so well it was turned into a library so that anyone could do the same kind of experiments.
How expensive is it to modify a code base so it uses Scientist to run a bunch of experiments? Which criteria have you applied at GitHub to optimize the cost/benefit ratio of using Scientist?
It depends on how expensive the parts of the code you want to run the experiment on are and how often that code is called. If the candidate code performs exactly the same as the control, then running the experiment will be 2x the cost. If the candidate code improves on the performance significantly, then the cost is much less.
We often slowly ramp up the percentage of requests that run a Scientist experiment if we think the experiment will be very expensive. Running an experiment on 1% or 5% of the traffic can be enough to gather a lot of performance and mismatch data.
Are there any additional features that you would like to have in Scientist?
We’ve built all of the features that we need so far, but we’re definitely interested in contributions if someone else finds a different way to use Scientist and wants to add features to support that.
Scientist requires Ruby 1.9 and a Unix box and can be installed through its gem.