Careful With Your Coverage Metrics

Christian Gruber takes some time to clarify the TDD stance on using code coverage metrics. He discusses what code coverage metrics do and don't tell you, how TDD fits into the picture, and how one might be best advised to use their code coverage metrics.

Code coverage for an application developed with good TDD will likely be very high (>80-90%). On the other hand, high code coverage on another app says little to nothing about whether that app was built by good TDD, or even TDD at all. Taken further, how well does high code coverage indicate your application is thoroughly tested?

Christian Gruber discusses this, prompted largely by another recent blog post by Kevin Pang on the subject. Out of the gates, Gruber's primary statement is that TDD proponents do not suggest code coverage as "a one true metric", that it is useful but only to degree and taken in context with other sources of feedback. He denounces Pang's claim that "(Pang) 100% code coverage has long been the ultimate goal of testing fanatics", stating that "(Gruber) high code coverage is a desired attribute of a well tested system, but the goal is to have a fully and sufficiently tested system".

He makes the following 6 assertions about code coverage, TDD, and "sufficient testing":

1. Code coverage is only meaningful in the context of well-written tests. It doesn’t save you from crappy tests.
2. Code coverage should only be measured on a line/branch if the covering tests are passing.
3. Code coverage suggests insufficiency, but doesn’t guarantee sufficiency.
4. Test-driven code will likely have the symptom of nearly perfect coverage.
5. Test-driven code will be sufficiently tested, because the author wrote all the tests that form, in full, the requirements/spec of that code.
6. Perfectly covered code will not necessarily be sufficiently tested.

Gruber then expands briefly on how TDD, being a design technique more than a testing tool, helps to provide thorough testing. He further asserts that "code coverage [in the context of TDD] is a great way to notice that you screwed up and missed something, but nothing else", a point that he and Pang seem to largely agree on.

Warning against the misuse of code coverage metrics is not a new thing, although it is a message that needs repeating as more and more organizations are taking on TDD for the first time (congrats!) and can easily fall into the "coverage as gospel" anti-pattern.

For more on this, also see the "Pragmatic Use of Code Coverage Analysis" section in a recent post by Jason Rudolph, which provides a good list of references to other experts' takes on this subject.