Many agile teams have used tools like Selenium and Watir for acceptance-testing or web user interface testing. By driving web browsers, these tools mirror the user's experience and offer strong support for testing dynamic interfaces that use DHTML and Ajax. However, as teams increasingly adopt tools like these, they find that a full suite of browser-driving web-interface tests can be time-consuming to run, causing build times to slow. Agile teams may then have to make some hard choices about comprehensive testing vs. build speed. Users of Selenium have had a limited ability to parallelize tests, either by running more than one test against a Remote Control (Selenium-RC), or by individually allocating tests against more than one Selenium-RC. These are sufficient for a small amount of parallelization, but are insufficient for massive paralleization. Selenium users now have another choice: Selenium Grid.
Selenium Grid is a free and open-source (Apache License 2.0) extension to Selenium which separates the concern of managing a grid of Selenium-RC instances from the concern of connecting to a Selenium-RC instance for the purpose of running a test. Essentially, Selenium Grid offers up a hub, which acts like a remote control for the purposes of a test, but transparently dispatches the test requests to one Selenium-RC instance from a grid of available instances on one or more machines. The Selenium Hub takes on these responsibilities:
- Allocating a Selenium Remote Control to a specific test (transparently).
- Limiting the number of concurrent test runs on a each Remote Control.
- Shielding the tests from the actual grid infrastructure.
Selenium Grid also allows the selenium tests to make specific, fine-grained environment choices by naming instances such that one test can be run on IE 7 on Windows XP, while another can target Firefox 1.5 on Ubuntu.
Ultimately, this allows an organization to build a sophisticated farm of test machines with as many environments as is necessary, and run the tests for one or more projects on that farm in a massively parallel way. This can result in significant improvements on test, and therefore build, time while reducing the per-project infrastructure. This is best realized in a large organization (such as Google, who uses a similar approach), but can be valuable even on a single machine for a single project.
Selenium Remote Control (Selenium-RC) has recently made strides towards improved performance even in a single-threaded environment, but long test runs could benefit significantly through increased use for multi-threaded and multi-machine testing. With sufficient test processing power and test isolation, it can be possible to reduce the testing time required to the length of the longest test.
Although Selenium Grid allows tests to remain agnostic to whether the test is being run on a single machine in series or across an entire farm, it is not responsible for running the tests in parallel, which can be accomplished through projects like TestNG, Parallel JUnit and DeepTest for Ruby.
InfoQ spoke with members of the Selenium Grid team about topics including the impact that running tests in parallel could have on the design of Selenium tests:
We discussed isolation and the challenges it presents before we started developing grid. We opted to leave the burden on the test writer for now, to design the tests such that they don't interfere with each other. It's an issue that existed prior to grid of course. You don't want the order in which your selenium tests run to have an effect on their success. In serial, a possible solution is to do some database setup before each test and cleanup afterwards. However, this is not a particularly elegant solution. Ideally you'd want your selenium tests to only know about the front end of the app, but pragmatically, by exposing a bit of the database to the test, it's faster and easier to write tests that exercise specific scenarios, and the tests run in less time due to less page navigation. It can become a bit philosophical, but regardless, we want the grid to be able to support either approach, and we're presently looking at ways to isolate tests at the database level. It's very alpha at this point, but it could at some point become an addition to grid or possibly a separate project.
Speaking about the future of Selenium Grid, the team suggested that some of these features could eventually find themselves in Selenum Grid:
- A more comprehensive management console
- Windows service behavior (and equivalent support for Solaris, Linux, etc.)
- Screensaver functionality (desktops can join the grid while idle)
- Support for using Amazon's EC2 (Electronic Compute Cloud) infrastructure as Selenium test farm.
ThoughtWorks offers support and consulting services for companies that wish to explore Selenium-RC and Selenium Grid in more detail.
For further reading, explore the Selenium Grid website (a formal project is being prepared at OpenQA) or read more about testing and unit testing here at InfoQ.
