Continuous Integration (CI) is a basic Extreme Programming (XP) practice, but it is also used by teams that would never consider XP for their work. Martin Fowler has pointed out that, at this point, it's become an essential part of any competent software development activity. In their book Continuous Integration: Improving Software Quality and Reducing Risk, Paul Duvall, Steve Matyas and Andrew Glover aim to help teams make this important practice a "non event" on their projects - something that happens unobtrusively and as a matter of course. When successfully implemented, CI ensures that any individual developer's work is only a few hours away from a shared project state and can integrated back into that state in minutes. InfoQ presents a pdf download (14MB) of Chapter 6: Continuous Testing, which presents strategies for writing the different kinds of tests required to ensure system quality.
Testing is a key area for CI improvement, since the bulk of an application's build time is usually applied to running tests. Poorly constructed test suites can cause build times to bog down, at which point teams start to circumvent their own agreed-upon CI practices just to recapture the time they need to code. These shortcuts increase the likelihood of "red" (broken) builds, and can lead to a "broken windows" scenario in which it's impossible to judge code quality because the build runs "red" more often than not, increasing the risk of serious production flaws, and provoking the addition of lengthy pre-release testing that delays deployment.
In this chapter on testing the authors describe the following topics and the relationships between them:
- Automate unit tests
Automate your unit tests, preferably with a unit testing framework such as NUnit or JUnit. These unit tests should have no external dependencies such as a file system or database. - Automate component tests
Automate your component tests with unit testing frameworks such as JUnit, NUnit, DbUnit, and NDbUnit if you are using a database. These tests involve more objects and typically take much longer to run than unit tests. - Automate system tests
System tests are longer to run than component tests and usually involve multiple components. - Automate functional tests
Functional tests can be automated using tools like Selenium (for Web applications) and Abbot for GUI applications. Functional tests operate from a user’s perspective and are typically the longest running tests in your automated test suite. - Categorize developer tests
By categorizing your tests into distinct “buckets,” you can run slower running tests (e.g., component) at different intervals than faster running tests (e.g., unit). - Run faster tests first
Run your unit tests prior to running component, system, and functional tests. You can achieve this by categorizing your tests. - Write tests for defects
Increase your code coverage by writing tests based on new defects and ensuring that the defect does not surface again. - Make component tests repeatable
Use database testing frameworks to make certain that the data is a “known state,” which helps make component tests repeatable.
Are you categorizing your automated tests, such as unit tests, component tests, system tests, and functional tests?The chapter is amply illustrated with examples from various testing frameworks, including TestNG, Ruby, DbUnit, StrutsTest, Selenium and JUnit.
Are you configuring your CI system to run each test category with different staged builds?
Are you writing automated unit tests for each defect?
How many asserts are in each of your test cases? Are you limiting each test case to one assert?
Are these tests automatable? Has your project committed automated developer tests to the version control repository?
Read the InfoQ exclusive excerpt: Chapter 6: Continuous Testing from the book Continuous Integration: Improving Software Quality and Reducing Risk by Paul Duvall with Steve Matyas and Andrew Glover. Have a look at the full table of contents on O'Reilly Safari for an idea of the other topics covered in the book.
