With the presence of high quality test generation tools like Agitar One and Parasoft's JTest, some are questioning the need to write tests manually. Uncle Bob (Martin) weighted in, exploring the weakness of the idea.
These tools are designed to examine an existing body of code. Based on branches, loops etc. they synthesize a set of observations about the code. The developer reviews the observations and chooses the relevant ones to create a set of test cases. In exploring legacy code this can be a very powerful way of understanding the existing behavior and creating a safety net before making changes.
However like any technology there are limits: these tools are only generating a set of observations about the code. They don't understand algorithms or the developer's intent. From Bob:
As a simple example of this, I have tried to generate tests for the bowling game program using two of the better known test generation tools. The interface to the Bowling Game looks like this:
public class BowlingGame {The idea is that you call roll each time the balls gets rolled, and you call score at the end of the game to get the score for that game.
public void roll(int pins) {...}
public int score() {...}
}The test generators could not randomly generate valid games. It's not hard to see why. A valid game is a sequence of between 12 and 21 rolls, all of which must be integers between 0 and 10. What's more, within a given frame, the sum of rolls cannot exceed 10. These constraints are just too tight for a random generator to achieve within the current age of the universe.
On the other hand, Test Driven Development (TDD), the process of writing the tests before writing the production code, works very differently. TDD works because it provides immediate feedback about the code the developer writes. Make a small change, run the test and the developer knows if the change was good. TDD ensures that the code matches our stated intentions (the tests). TDD makes us think about the design of the code from the point of view the consumer and not just the author. It makes us think about every conditional and loop. Additionally, as James Carr mentions, TDD forces us to consider how tightly coupled our code is. From Bob again:
Using a test generator breaks this concept because the generator writes the test using the production code as input. The generated test is not a human restatement, it is an automatic translation. The human states intent only once, and therefore does not gain insights from restatement, nor does the generated test check that the intent of the code was achieved. It is true that the human must verify the observations, but compared to TDD that is a far more passive action, providing far less insight into defects, design and intent.
... This does not mean that test generators aren't useful. ... I think they can help to partially characterize a large base of legacy code.
