Property-based testing is an approach that involves specifying statements that should always be true, rather than relying on specific examples. It enables you to test functions across a large number of inputs with fewer tests.
Lucy Mair spoke about property-based testing at NDC Oslo 2023.
Property-based testing is a technique where one identifies properties – statements of the form "for all inputs that satisfy some precondition, some predicate holds" – and writes their tests using those, as Mair explained:
First, you write a function that specifies your property; it takes inputs that satisfy the precondition, then asserts the predicate holds. Next, you use a property-based testing framework which will generate many different inputs and run your function to check the predicate holds.
Mair presented examples using FsCheck and F#, but this technique can be used for any language, they said.
The most immediate benefit to writing property-based tests is being able to test functions across a large number of inputs without having to write a large number of tests, Mair said. Every run of a property-based test will use different inputs, which can give you confidence your code works in a general case. Mair mentioned that this is particularly valuable in finding edge cases for inputs you may never have thought to test.
Property-based tests work by generating and testing many cases so will take longer than traditional unit tests. The performance implications will vary based on your particular use case, Mair mentioned.
InfoQ interviewed Lucy Mair about property-based testing.
InfoQ: Can you give an example of a property?
Lucy Mair: Properties are related to functions, so let’s consider the case of a function concat(a, b) that concatenates two lists. As an example, concat([0, 1, 2], [3, 4]) = [0, 1, 2, 3, 4]
What statements can we write about c = concat(a, b) that are always true?
- The length of c will be a.length + b.length.
- All items within a will be present in c.
- All items within b will be present in c.
- Concatenating with an empty list returns the same list a = concat(a, []) = concat([], a).
InfoQ: How can we do property-based testing in front-end development?
Mair: Front-end development in JavaScript presents some unique challenges on the property-based testing front.
One issue is that JavaScript’s relatively lax runtime typing means generating an object of the correct shape for running the tests is non-trivial. TypeScript doesn’t help us here either; TypeScript types are compile-time only so cannot be used to create correctly shaped objects.
The second issue is that rendering components for front-end tests is slow, particularly if the test involves simulating user interactions. If your property-based tests include rendering they are likely to be unusably slow.
InfoQ: What are the potential downsides to property-based testing?
Mair: For property-based tests on pure functions, a traditional unit test may take e.g. 10 ms, and a property-based test may take e.g. 200 ms, so the performance impact is negligible. However, if your test is doing "slow" things, such as rendering a component, making API calls, or performing compute-intensive tasks then property-based tests may have too much of a performance overhead to use.
Introducing property-based tests to an existing code base may be a challenge. You are likely to have edge cases that may never arise during the running of the program and as such are never tested. An example of this is dividing by a number - zero is an invalid number in this case but you may know the number to never be non-zero due. This represents a specification that should be codified in tests but rarely is in practice.