Smashtest: New Web App Testing DSL Focusing on Speed

Peter Tesler, former lead software development engineer in Test (SDET) at Walt Disney, recently announced the first major release of Smashtest, a new domain-specific language for describing and deploying test cases. Smashtest strongly focuses on improving tester productivity by expressing succintly in the language the repetitive testing chores performed by developers. The new language comes with dedicated tooling including a test runner which can run tests in parallel, and a REPL.

The major novel feature of Smashtest consists of allowing testers to declaratively describe their tests in a tree-like format. The following Smashtest script given on the Smashtest site:

Open Chrome
Open Firefox
Open Safari

    Navigate to 'site.com'

        Click 'Sign In'

            Type {username:} into 'username box'

                {username} is 'joe'
                {username} is 'bob'
                {username} is 'mary'

                    Verify success

                {username} is 'baduser'

                    Verify error

describes 3 (Chrome, Firefox and Safari browsers) x (3 + 1) = 12 tests in one go, covering both happy paths (joe, bob, mary) and error paths (baduser) in the same description:

Smashtest is thus a whitespace-sensitive language, using the white space to organize the tests into a tree. The tests to execute are deduced by depth-first exploration of the tree. Smashtest is best used with editors which can respect the existing spaces. A plug-in for the Atom Editor is made available in the release.

The Smashtest language includes a mocking API, allowing in particular to mock time and geolocation. Network conditions and throttling can also be described in the language (feature only available on Chrome browser). The language also includes an API for finding elements on a page, on which to conduct assertions.

The Smashtest Command Line Interface (CLI) includes tooling for Continuous Integration/Continuous Delivery and a REPL(Read–Eval–Print Loop) interpreter. The CLI can additionally create reports with automatically-generated screenshots:

InfoQ interviewed Peter Tesler, creator of Smashtest, about Smashtest’s value proposition:

InfoQ: What drove you to write Smashtest? Why a new testing language?

Peter Tesler: The speed at which we’re able to write, deploy, update, and tear down automated tests is of utmost importance. Software changes rapidly and tests need to keep up. I wanted to build a testing language that made speed a priority.
I also wanted it to reflect how we think when we test, namely breadth-first. Meaning, when I’m at a particular state in my app, I think of all the permutations that can happen next. If I’m at a login box, for example, I list inside my head all the potential usernames I can enter. Users think depth-first, or in a linear series of steps. Traditional tests have always reflected the latter, but reflecting the former (writing tests as a tree of permutations) makes it easier to write as you think, see what you have covered, and do so quickly (since there’s much less to type out).

InfoQ: How does Smashtest compare with Gherkin-based solutions organized around Feature/Scenario and Given/When/Then triples?

Tesler: Smashtest handles Gherkin too. But traditional BDD/TDD tools don’t store tests as trees.
We also made optional what we thought was unnecessary, such as describing or naming tests, outside of the steps themselves. Tests have a tendency to look very similar to other tests, and usually get called unhelpful names like ‘cart test 014’. Having to come up with names only slows you down (though you can still add names and descriptions if you want to).

InfoQ: You mention an increase in testing speed. What parts of the testing process does Smashtest refer to?

Tesler: Test writing and maintenance. The time it takes to write a functional automated test these days is incredibly long, especially when we’re talking about selenium webdriver tests. The longer it takes to churn out a new test, and the more often the app and/or requirements change, the less bang for the buck you get out of automation. To the point where many organizations give up on automation altogether, in favor of manual testing (where test creation cost is low).

InfoQ: Do you have empirical or hard measures of speed improvements vs. hand-made or existing testing solutions?

Tesler: While we don’t have concrete empirical measures yet, I did time myself writing a few sample test scenarios using traditional webdriverjs vs. Smashtest, and saw around a 10-fold improvement. It’s really easy to add a single line in Smashtest and instantly double the permutations you have covered.

InfoQ: Three reasons for developers, testers and product owners to adopt Smashtest?

Tesler:

1. Speed of writing tests. Ease of reading them.
2. It handles a lot right out-of-the-box. UI and API testing, various kinds of mocking, even a language to describe what a page should look like (so you don’t need 20+ verify steps, one after another).
3. Live reports. They update as the tests run, combine similar failures into groups (makes it much easier to debug), display all the screenshots/logs, and they’re downright beautiful!

InfoQ: Three reasons for developers, testers and product owners not to adopt Smashtest?

Tesler:

1. Like with any language, there’s a learning curve.
2. You’ll have to break with tradition and write tests in a new way.
3. Right now it has built-in support for testing web UIs (with selenium webdriver) and HTTP APIs. If you want to test anything else, you’ll have to write some custom js. But anything that can run in nodejs can run in Smashtest.

InfoQ: What comes next for Smashtest?

Tesler: Right now we’re focused on spreading the word and building the community. We’ll probably add support for more editors and languages, depending on where there’s demand.