A common theme amongst people professing "best practices" for unit tests is that you should only write a single assertion, maybe two, for each test. People who make these proclamations rarely show any unit test and those that do only show one. Yet if you follow their advice, a dozen other unit tests may be required to ensure quality for even a trivial operation. This article intends to demonstrate, by example, that multiple assertions per test are both necessary and beneficial.
Let us consider a fairly typical Person object that you would find in a data-binding scenario:
Testing FirstName
The first thing we want to test is setting the FirstName property. So to begin:
[TestMethod]
public void Person_FirstName_Set()
{
varperson = new Person("Adam", "Smith");
person.FirstName = "Bob";
Assert.AreEqual("Bob", person.FirstName);
}
Next we want to test the property changed notification for first name.
[TestMethod]
public void Person_FirstName_Set_PropertyChanged()
{
var person = new Person("Adam", "Smith");
var eventAssert = new Granite.Testing.PropertyChangedEventAssert(person);
person.FirstName = "Bob";
eventAssert.Expect("FirstName");
}
When we run this test we will get a failure with the message "expected PropertyName ‘FirstName’ but received ‘IsChanged’". Apparently setting the first name property tripped the IsChanged flag and we need to account for it. So we add that into the mix:
[TestMethod]
public void Person_FirstName_Set_PropertyChanged()
{
var person = new Person("Adam", "Smith");
var eventAssert = new Granite.Testing.PropertyChangedEventAssert(person);
person.FirstName = "Bob";
eventAssert.SkipEvent(); //this was IsChanged
eventAssert.Expect("FirstName");
}
With two passing tests, we consider what else needs to change when FirstName is altered. Looking over the API, the properties IsChanged and FullName took promising.
[TestMethod]
public void Person_FullName_Changed_By_Setting_FirstName()
{
var person = new Person("Adam", "Smith");
person.FirstName = "Bob";
Assert.AreEqual("Bob Smith", person.FullName);
}
[TestMethod]
public void Person_IsChanged_Changed_By_Setting_FirstName()
{
var person = new Person("Adam", "Smith");
person.FirstName = "Bob";
Assert.IsTrue(person.IsChanged);
}
And of course if these properties have changed we need to capture their property change notifications:
[TestMethod]
public void Person_IsChanged_Property_Change_Notification_By_Setting_FirstName()
{
var person = new Person("Adam", "Smith");
var eventAssert = new PropertyChangedEventAssert(person);
person.FirstName = "Bob";
eventAssert.Expect("IsChanged");
}
[TestMethod]
public void Person_FullName_Property_Change_Notification_By_Setting_FirstName()
{
var person = new Person("Adam", "Smith");
var eventAssert = new PropertyChangedEventAssert(person);
person.FirstName = "Bob";
eventAssert.SkipEvent(); //this was IsChanged
eventAssert.SkipEvent(); //this was FirstName
eventAssert.Expect("FullName");
}
Our next two tests address the HasErrors property and the ErrorsChanged event.
[TestMethod]
public void Person_FirstName_Set_HasErrorsIsFalse()
{
var person = new Person("Adam", "Smith");
person.FirstName = "Bob";
Assert.IsFalse(person.HasErrors);
}
[TestMethod]
public void Person_FirstName_Set_ErrorsChanged_Did_Not_Fire()
{
var person = new Person("Adam", "Smith");
var errorsChangedAssert = new ErrorsChangedEventAssert(person);
person.FirstName = "Bob";
errorsChangedAssert.ExpectNothing();
}
We have eight tests so far, which means we have accounted for everything that should change when we alter the FirstName property. But that doesn’t mean we are done yet. We also need to make sure nothing else was changed accidentally. In theory that means several more assertions and an equal number of tests, but we’ll cheat and replace the HasErrors test with a utility called ChangeAssert.
[TestMethod]
public void Person_FirstName_Set_Nothing_Unexpected_Changed()
{
var person = new Person("Adam", "Smith");
var changeAssert = new ChangeAssert(person);
person.FirstName = "Bob";
changeAssert.AssertOnlyChangesAre("FirstName", "FullName", "IsChanged");
}
ChangeAssert simply captures the state of the object using reflection. Then later you can assert that nothing changed expect the specific properties you indicate.
Congratulations, you are done with your first test case. One down, many, many more to go.
What Do You Mean "One" Test Case?
The eight tests that we ended up keeping only cover the scenario in which FirstName is changed from "Adam" to "Bob" when all other fields are not in an error state and LastName is not null or empty. Let’s look at a more complete list of test cases:
- Setting the FirstName to "Adam"
- Setting the FirstName to null
- Setting the FirstName to an empty string
- Cases 1 thru 3, but when LastName is null.
- Cases 1 thru 3, but when LastName is an empty string.
- Cases 1 thru 5, but with FirstName starting with null.
- Cases 1 thru 5, but with FirstName starting with an empty string.
So far we are looking at 27 different scenarios. With up to 8 individual test per scenario we get an upper bound of 216 tests just for this one property. And in the grand scheme of things, this is a fairly trivial piece of code. So what are we to do?
Tests Can Have Code Smells Too
Looking back at our eight tests for the first test case, we see that each test has exactly the same setup and exactly the same operation. The only difference is the assertions we wrote. That’s what is known in the industry as a "Code smell". In fact there are two code smells according to the list in Wikipedia:
- Duplicated code
- Excessively long identifiers
We can eliminate both of these code smells simply by combining the assertions into one test:
[TestMethod]
public void Person_FirstName_Set()
{
var person = new Person("Adam", "Smith");
var eventAssert = new PropertyChangedEventAssert(person);
var errorsChangedAssert = new ErrorsChangedEventAssert(person);
var changeAssert = new ChangeAssert(person);
person.FirstName = "Bob";
Assert.AreEqual("Bob", person.FirstName, "FirstName setter failed");
Assert.AreEqual("Bob Smith", person.FullName, "FullName not updated with FirstName changed");
Assert.IsTrue(person.IsChanged, "IsChanged flag was not set when FirstName changed");
eventAssert.Expect("IsChanged");
eventAssert.Expect("FirstName");
eventAssert.Expect("FullName");
errorsChangedAssert.ExpectNothing("Expected no ErrorsChanged events");
changeAssert.AssertOnlyChangesAre("FirstName", "FullName", "IsChanged");
}
Since it is still important to know what causes a test to fail, we make sure the assertions include a message.
Unit Tests and Code Reuse
Looking back over our 27 test cases, we may decide that setting the first name to null or empty string should have the same net effect. So we generalize it:
[TestMethod]
public void Person_FirstName_Set_Empty()
{
Person_FirstName_Set_Invalid(String.Empty);
}
[TestMethod]
public void Person_FirstName_Set_Null()
{
Person_FirstName_Set_Invalid(null);
}
public void Person_FirstName_Set_Invalid(string firstName)
{
var person = new Person("Adam", "Smith");
var eventAssert = new PropertyChangedEventAssert(person);
var errorsChangedAssert = new ErrorsChangedEventAssert(person);
var changeAssert = new ChangeAssert(person);
Assert.IsFalse(person.IsChanged, "Test setup failed, IsChanged is not false");
Assert.AreEqual("Adam", person.FirstName, "Test setup failed, FirstName is not Adam");
Assert.AreEqual("Smith", person.LastName, "Test setup failed, LastName is not Smith");
person.FirstName = firstName;
Assert.AreEqual(firstName , person.FirstName, "FirstName setter failed");
Assert.AreEqual("Smith", person.FullName, "FullName not updated with FirstName changed");
Assert.IsTrue(person.IsChanged, "IsChanged flag was not set when FirstName changed");
eventAssert.Expect("IsChanged");
eventAssert.Expect("FirstName");
eventAssert.Expect("FullName");
Assert.IsTrue(person.HasErrors, "HasErrors should have remained false");
errorsChangedAssert.ExpectCountEquals(1, "Expected an ErrorsChanged event");
changeAssert.AssertOnlyChangesAre("FirstName", "FullName", "IsChanged", "HasErrors");
}
Seeing the differences between Person_FirstName_Set and Person_FirstName_Set_Invalid are fairly minor, we may seek to generalize it further.
[TestMethod]
public void Person_FirstName_Set_Valid()
{
Person_FirstName_Set("Bob", false);
}
[TestMethod]
public void Person_FirstName_Set_Empty()
{
Person_FirstName_Set(String.Empty, true);
}
[TestMethod]
public void Person_FirstName_Set_Null()
{
Person_FirstName_Set(null, true);
}
public void Person_FirstName_Set(string firstName, bool shouldHaveErrors)
{
var person = new Person("Adam", "Smith");
var eventAssert = new PropertyChangedEventAssert(person);
var errorsChangedAssert = new ErrorsChangedEventAssert(person);
var changeAssert = new ChangeAssert(person);
Assert.IsFalse(person.IsChanged, "Test setup failed, IsChanged is not false");
Assert.AreEqual("Adam", person.FirstName, "Test setup failed, FirstName is not Adam");
Assert.AreEqual("Smith", person.LastName, "Test setup failed, LastName is not Smith");
person.FirstName = firstName;
Assert.AreEqual(firstName, person.FirstName, "FirstName setter failed");
Assert.AreEqual((firstName + " Smith").Trim(), person.FullName, "FullName not updated with FirstName changed");
Assert.AreEqual(true, person.IsChanged, "IsChanged flag was not set when FirstName changed");
eventAssert.Expect("IsChanged");
eventAssert.Expect("FirstName");
eventAssert.Expect("FullName");
if (shouldHaveErrors)
{
Assert.IsTrue(person.HasErrors, "HasErrors should have remained false");
errorsChangedAssert.ExpectCountEquals(1, "Expected an ErrorsChanged event");
changeAssert.AssertOnlyChangesAre("FirstName", "FullName", "IsChanged", "HasErrors");
}
else
{
errorsChangedAssert.ExpectNothing("Expected no ErrorsChanged events");
changeAssert.AssertOnlyChangesAre("FirstName", "FullName", "IsChanged");
}
}
There is definitely a limit as to how much you can generalize your test code before it becomes confusing. However, a meaningful test name paired with a good description on each assertion will go a long way towards keeping your tests understandable.
Controlling the Variables
So far all of our assertions considered only the outcomes of the test case. They assumed each person object started in a known-good state and went from there. But if we want to be scientific in our testing we need to ensure that we "control the variables". Or in other words, we need to ensure that the state of the world actually matches what we expect.
This leads to the following set of assertions.
Assert.IsFalse(person.HasErrors, "Test setup failed, HasErrors is not false");
Assert.IsFalse(person.IsChanged, "Test setup failed, IsChanged is not false");
Assert.AreEqual("Adam", person.FirstName, "Test setup failed, FirstName is not Adam");
Assert.AreEqual("Smith", person.LastName, "Test setup failed, LastName is not Smith");
Since we don’t want to repeat those assertions at the beginning of each test, we may choose to move it into a factory method that we can trust to always give us a clean object. This is especially useful if we end up reusing the same setup for testing other properties.
[TestMethod]
public void Person_FirstName_Set()
{
var person = GetAdamSmith();
...
Table Style Testing
What it really comes down to is the number of "test methods" you have really has no bearing on how comprehensive your tests are. They are just a convenient way to organize and run your test cases.
Another way to organize a lot of test cases is to use a single table-driven test method. You lose the ability to run individual tests, but you gain the ability to add new test cases with a single line of code. The "table" in table style testing may come from a XML file, a database table, hard coded in arrays, or simply be the same function repeated called with different values. Some frameworks such as MBTest even allow you to specify the test cases using attributes, but to keep the examples portable we’ll stick to the lowest common denominator.
[TestMethod]
public void Person_FullName_Tests()
{
Person_FullName_Test("Bob", "Jones", "Bob Jones");
Person_FullName_Test("Bob ", "Jones", "Bob Jones");
Person_FullName_Test(" Bob", "Jones", "Bob Jones");
Person_FullName_Test("Bob", " Jones", "Bob Jones");
Person_FullName_Test("Bob", "Jones ", "Bob Jones");
Person_FullName_Test(null, "Jones", "Jones");
Person_FullName_Test(string.Empty, "Jones", "Jones");
Person_FullName_Test(" ", "Jones", "Jones");
Person_FullName_Test("Bob", "", "Bob");
Person_FullName_Test("Bob", null, "Bob");
Person_FullName_Test("Bob", string.Empty, "Bob");
Person_FullName_Test("Bob", " ", "Bob");
}
private void Person_FullName_Test(string firstName, string lastName, string expectedFullName)
{
var person = GetAdamSmith();
person.FirstName = firstName;
person.LastName = lastName;
Assert.AreEqual(expectedFullName, person.FullName,
string.Format("Incorrect full name when first name is '{0}' and last name is '{1}'"
firstName ?? "<NULL>", lastName ?? "<NULL>"));
}
It is very important that you use error messages that include the parameters when using this technique. If you don’t, you’ll find yourself having to step through code when trying to determine which particular combination isn’t correct.
Conclusion
When writing units of any variety one should try to maximize these factors:
- Meaningful test coverage per unit of effort
- Maintainability in the face of a changing code base
- Performance of the test suite
- Clarity in terms of what is being tested and why
While these factors are often in conflict, the careful use of multiple assertions per test can serve to improve all four factors by:
- Reducing the amount of boilerplate code to be written
- Reducing the amount of boilerplate code to be updated when the API changes
- Reducing the physical number of boilerplate code to be executed per assertion
- Documenting in one place all of the assertions that apply to a given operation
About the Author
Jonathan Allen has been writing news report for InfoQ since 2006 and is currently the lead editor for the .NET queue. If you are interested in writing news or educational articles for InfoQ please contact him at jonathan@infoq.com.
