A small team of dedicated developers is currently working on JUnit 5, the next version of one of Java’s most popular libraries. While the improvements on the surface are deliberately incremental, real innovation happens under the hood, with the potential to redefine testing on the JVM.

After a prototype in November 2015 and an alpha version in February 2016, July saw the releases of Milestone 1 and Milestone 2. And we’re here to test drive the latest and greatest!

In this first part we will find out how to start writing tests, have a look at all the little improvements the new version brings, discuss why the JUnit team decided it was time for a rewrite, and—to top it off—see how the new architecture can be a game changer for testing on the JVM.

The second part will look in more detail at how to run tests, show off some of JUnit’s cool new features, and demonstrate how the core functionality can be extended.

Writing Tests

Let's start in medias res and have a look at how you can quickly hammer out some tests.

Five Second Setup

We will have a more detailed look at the setup and architecture later. For now we simply import these artifacts with our build tool of choice:

org.junit.jupiter:junit-jupiter-api:5.0.0-M2
org.junit.jupiter:junit-jupiter-engine:5.0.0-M2
org.junit.platform:junit-platform-runner:1.0.0-M2

This is today; once full JUnit 5 support comes around, we will only need the junit-jupiter-api artifact. (More on that later.)

Next we need a class to contain our tests:

package com.infoq.junit5;

import org.junit.platform.runner.JUnitPlatform;
import org.junit.runner.RunWith;

@RunWith(JUnitPlatform.class)
public class JUnit5Test {
    // tests go here
}

Now we're all set up to write JUnit 5 tests and have our IDE or build tool execute them.

A Simple Example

For simple tests not much has changed:

@RunWith(JUnitPlatform.class)
public class JUnit5Test {

    @BeforeAll
    static void initializeExternalResources() {
   	 System.out.println("Initializing external resources...");
    }

    @BeforeEach
    void initializeMockObjects() {
   	 System.out.println("Initializing mock objects...");
    }

    @Test
    void someTest() {
   	 System.out.println("Running some test...");
   	 assertTrue(true);
    }

    @Test
    void otherTest() {
   	 assumeTrue(true);

   	 System.out.println("Running another test...");
   	 assertNotEquals(1, 42, "Why wouldn't these be the same?");
    }

    @Test
    @Disabled
    void disabledTest() {
   	 System.exit(1);
    }

    @AfterEach
    void tearDown() {
   	 System.out.println("Tearing down...");
    }

    @AfterAll
    static void freeExternalResources() {
   	 System.out.println("Freeing external resources...");
    }

}

On the surface JUnit 5 has deliberately undergone just an incremental improvement; the more revolutionary features are under the covers, and we will discuss those in a minute. But first let's look at some of the more obvious improvements in this version.

Improvements

Visibility

The most obvious change is that test methods no longer must be public. Package visibility suffices (but private does not), so we can keep our test classes free from the clutter of lots of public keywords.

Theoretically test classes can have default visibility as well. But because of the simple setup we just did, our tools will only scan public classes for annotations. This will change once JUnit 5 support comes around.

Test Lifecycle

@Test

The most basic JUnit 4 annotation is @Test, used to mark methods that are to be run as tests.

The annotation is virtually unchanged, although it no longer takes optional arguments; expected exceptions can now be verified via assertions. (For timeouts there is not yet a replacement.)

Before And After

To run code to set up and tear down our tests we can use @BeforeAll, @BeforeEach, @AfterEach, and @AfterAll. They are more appropriately named but semantically identical to JUnit 4’s @BeforeClass, @Before, @After, and @AfterClass.

Because a new instance is created for each test, and @BeforeAll / @AfterAll are only called once for all of them, it is not clear on which instance they should be invoked so they have to be static (as was the case with @BeforeClass and @AfterClass in JUnit 4).

If different methods are annotated with the same annotation, the execution order is deliberately undefined.

Disabling Tests

Tests can simply be disabled with @Disabled, which is equivalent to JUnit 4's @Ignored. This is just a special case of a Condition, which we will see later when we talk about extending JUnit.

Assertions

After everything was set up and executed, it is ultimately up to assertions to verify the desired behavior. There have been a number of incremental improvements in that area as well:

• Assertion messages are now last on the parameter list. This makes calls with and without messages more uniform as the first two parameters are always the expected and actual value, and the optional argument comes last.
• Using lambdas, assert messages can be created lazily, which can improve performance if creation is a lengthy operation.
• Boolean assertions accept predicates.

Then there is the new assertAll, which checks a group of related invocation results and, if the assertion fails, does not short-circuit but prints values for all of them:

@Test
void assertRelatedProperties() {
    Developer dev = new Developer("Johannes", "Link");

    assertAll("developer",
   		 () -> assertEquals("Marc", dev.firstName()),
   		 () -> assertEquals("Philipp", dev.lastName())
    );
}

This yields the following failure message:

org.opentest4j.MultipleFailuresError: developer (2 failures)
    expected: <Marc> but was: <Johannes>
    expected: <Philipp> but was: <Link>

Note how the printout includes the last name failure even after the first names had already failed the assertion.

Finally we have assertThrows and expectThrows, both of which fail the test if the specified exception is not thrown by the called method. But to further assert properties of the exception (e.g. that the message contains certain information), expectThrows returns it.

@Test
void assertExceptions() {
    // assert that the method under test
    // throws the expected exception */
    assertThrows(Exception.class, unitUnderTest::methodUnderTest);

    Exception exception = expectThrows(
        Exception.class,
        unitUnderTest::methodUnderTest);
    assertEquals("This shouldn't happen.", exception.getMessage());
}

Assumptions

Assumptions make it possible to run tests if certain conditions are as expected. An assumption must be phrased as a boolean expression, and if the condition is not met the test exits. This can be used to reduce the run time and verbosity of test suites, especially in the failure case.

@Test
void exitIfFalseIsTrue() {
    assumeTrue(false);
    System.exit(1);
}

@Test
void exitIfTrueIsFalse() {
    assumeFalse(this::truism);
    System.exit(1);
}

private boolean truism() {
    return true;
}

@Test
void exitIfNullEqualsString() {
    assumingThat(
             // state an assumption (a false one in this case) ...
   		 "null".equals(null),
             // … and only execute the lambda if it is true
   		 () -> System.exit(1)
    );
}

Assumptions can either be used to abort tests whose preconditions are not met (assumeTrue and assumeFalse) or to execute specific parts of a test when a condition holds (assumimgThat). The main difference is that aborted tests are reported as disabled, whereas a test that was empty because a condition did not hold shows green.

A Little History

As we have seen, JUnit 5 comes with a number of incremental improvements to how we will write tests. Of course it also brings new features to the table, which we will look at later. But, interestingly enough, the real reason that so much effort was poured into a new version was somewhat deeper.

Why Rewrite JUnit?

JUnit And The Tools

With development techniques like test-driven development and continuous integration becoming ever more widespread, tests have become increasingly more important for a developer's day to day business. As such the requirements for IDEs grew as well. Developers wanted simple and specific execution (down to individual methods), quick feedback, and easy navigation. Build tools and CI servers added their own requirements.

How was JUnit 4 prepared for this? As it turns out, not well. Besides its lone dependency, Hamcrest 1.3, JUnit 4.12 is a monolithic artifact containing the API developers write tests against and the engine running these tests, and that’s it. Discovering tests, for example, had to be implemented again by every tool that wanted to do that.

Unfortunately this was not sufficient to support some advanced tool features. Tool developers often had to improvise by using reflection to access JUnit’s internal APIs, non-public classes, and even private fields. Suddenly implementation details that could otherwise be freely refactored, became de facto parts of the public API. The resulting lock-in made maintenance unpleasant and further improvements difficult.

Johannes Link, initiator of the current rewrite, called it a "Sisyphean struggle" and summarizes the situation as follows:

The success of JUnit as a platform prevents the development of JUnit as a tool.

Extending JUnit

Test runners were the original mechanism to extend JUnit 4. We could create our own Runner implementation and tell JUnit to use it by annotating our test class with @RunWith(OurNewRunner.class). Our custom runner would have to implement the complete test life cycle, including instantiation, setup and teardown, running the test, handling exceptions, sending notifications, etc.

This made it rather heavyweight and inconvenient for creating small extensions. And it had the severe limitation that there could always only be one runner per test class, which made it impossible to combine them and benefit from features of, e.g., the Mockito and the Spring runners at the same time.

To alleviate these limitations, JUnit 4.7 introduced rules. The default JUnit 4 runner would wrap tests as a Statement and pass it to rules that were applied to that test. They could then do things like create a temporary folder, run the test in Swing’s event dispatch thread, or let the test time out if it ran too long.

Rules were a big improvement but are generally limited to executing some code before, during, and after a test was run; but outside of those lifecycle points, little support was available for implementing more demanding extensions.

Then there is the fact that all test cases have to be known before their execution starts. This prevents the dynamic creation of test cases, e.g. in response to observed behaviour during test execution.

And now there were two competing extension mechanisms, each with its own limitations but also with quite an overlap. This made clean extension difficult. Additionally, composing different extensions has been reported to be problematic and would often not perform as expected.

Call Of The Lambda

JUnit 4 is over 10 years old and still uses Java 5, so it missed out on all of the subsequent Java language improvements, most notably lambda expressions, which would allow constructs such as:

   test(“someTest”, () -> {
   	 System.out.println("Running some test...");
   	 assertTrue(true);
    });

Enter JUnit Lambda

So we now understand the bigger picture leading to the idea of a rewrite, and in 2015 the JUnit Lambda team formed around this goal. At its core were Johannes Link (who subsequently left the project), Marc Philipp, Stefan Bechtold, Matthias Merdes, and Sam Brannen.

Sponsoring And Crowd-Funding

It is interesting to note that Andrena Objects, Namics, and Heidelberg Mobil, the employers of Marc Philipp, Stefan Bechtold, and Matthias Merdes, respectively, magnanimously sponsored six weeks of full-time work each on the project. It was clear however that additional development time and more funding would be required to organize an initial workshop. They estimated that they'd need at least €25,000 and started a crowdfunding campaign on Indiegogo with that goal. After a somewhat slow start things ultimately accelerated and resulted in a whopping €53,937 (around US$60,000).

This allowed the team to spend about two months of full-time development on the project. By the way, the use of those funds is fully transparent.

Prototype, Alpha Version, and Milestones

In October 2015 the JUnit Lambda team kicked off with a workshop in Karlsruhe, Germany before embarking on a month of full-time work. The resulting prototype was released four weeks later, demonstrating many of the new features, and even some experimental ones that didn't make it into the current version.

After collecting feedback the team began work on the next version, rebranded as JUnit 5, and released the alpha version in February 2016. Another round of feedback and five months of intense development work later, Milestone 1 was released July 7th, 2016. Two weeks and a couple of bug-fixes later, Milestone 2, our current subject, saw the light of day. In June the project underwent another transformation and JUnit 5 got split into JUnit Jupiter, JUnit Platform, and JUnit Vintage, a distinction we will discuss next.

Feedback

With a new version out in the wild, the project is collecting feedback once again.The community is urged to try JUnit 5, and open issues and pull-requests on GitHub. We should seize this opportunity and weigh in!

The team and some early adopters also started to give talks about JUnit 5. The next ones are:

• SpringOne Platform in Las Vegas, NV; August 2nd, 2016

• JavaZone 2016 in Oslo, Norway; September 7th, 2016

• JavaOne in SanFrancisco, CA; September 18th, 2017

• JAX London 2016 in London, UK; October 10th, 2016

Next Milestones And Final Version

Having spent the crowd's funds a couple of months ago the team went back to their day jobs, working on JUnit 5 in their free time. And they are making good progress! Work on Milestone 3 is already underway and its release is planned for later this year. And who knows, maybe it will already be the final version.

Architecture

We have seen how JUnit 4's monolithic architecture made development difficult.

So how is the new version going to change that?

Separating Concerns

A test framework has two important tasks:

• enabling developers to write tests
• enabling tools to run tests

When pondering the second point for a while it becomes obvious that it contains parts that are identical across different test frameworks. Whether JUnit, TestNg, Spock, Cucumber, ScalaTest, etc., tools typically need a test's name and result, a way to execute tests, they are interested in their reporting hierarchy, etc.

Why repeat code that handles these points across different frameworks? Why require tools to implement specific support for this or that framework (and version) if, on an abstract level, the features are always the same?

JUnit As A Platform

JUnit may be the most used Java library and is surely the most popular testing framework on the JVM. This success goes hand in hand with a tight integration in IDEs and build tools.

At the same time, other testing frameworks are exploring interesting new approaches to testing, albeit that the lack of integration often draws developers back to JUnit. Maybe they could benefit from JUnit's success and piggy-back on the integration that is provided for it? (Much like so many languages benefit from Java's success by leveraging the JVM.)

Migration

But this is not only a theoretical argument; it is important for the JUnit project itself because it connects to the critical question of migration. Should existing and even new tools support versions 4 and 5 in parallel? It might be hard to persuade tool vendors to add so much code, but if they don't, developers would have no incentive to upgrade their testing framework.

If JUnit 5 could run both versions of the tests behind a uniform API, that would clearly be more powerful and convenient, allowing tools to remove the obsolete JUnit 4 integration.

Modularization

These thoughts lead to a decoupled architecture, where different roles (developers, runtimes, tools) rely on different artifacts:

1. an API for developers to write tests against
2. an engine for each API to discover, present and run the corresponding tests
3. an API that all engines have to implement so they can be used uniformly
4. a mechanism that orchestrates the engines

This separates “JUnit the tool” (1. and 2.) from “JUnit the platform” (3. and 4.). To make this distinction clearer, the project chose to endow it with a naming schema:

• The new API we have seen above (and will continue to discuss in the next article) is called JUnit Jupiter. It’s what us developers will have the most contact with.
• The platform for tools will fittingly be called JUnit Platform.
• We have not seen it yet, but there is also a JUnit Vintage subproject, which will adapt JUnit 3 and 4 test to be run by JUnit 5.

JUnit 5 is the sum of these three parts. And its new architecture is the result of that distinction:

junit-jupiter-api (1)

The API against which developers write tests. Contains the annotations, assertions, etc. that we saw earlier.

junit-jupiter-engine (2)

An implementation of the junit-engine-api (see below) that runs JUnit 5 tests, i.e. those written against junit-jupiter-api.

junit-platform-engine (3)

The API all test engines have to implement, so they are accessible in a uniform way. Engines might run typical JUnit tests or could opt to run tests written with TestNG, Spock, Cucumber, etc. They can make themselves available to the launcher (see below) by registering themselves with Java's ServiceLoader.

junit-platform-launcher (4)

Uses the ServiceLoader to discover test engine implementations and orchestrate their execution. It provides an API for IDEs and build tools to interact with test execution, for example by launching individual tests and showing their results.

The benefits of this architecture are immediately apparent; we just need two more components to make it execute JUnit 4 tests as well:

junit-4.12 (1)

The JUnit 4 artifact acts as the API that the developer implements her tests against, but also contains the main functionality of how to run the tests.

junit-vintage-engine (2)

An implementation of the junit-platform-engine that runs tests written with JUnit 4. It could be seen as an adapter of JUnit 4 for version 5.

Other frameworks already provide the API to write tests against, so all that is missing for full JUnit 5 integration is a test engine implementation.

A picture is worth a thousand words:

API Lifecycle

Next problem to solve was all of those internal APIs everybody was using. The team therefore created a lifecycle for its API. Here it is, with the explanations straight from the source:

Internal

Must not be used by any code other than JUnit itself. Might be removed without prior notice.

Deprecated

Should no longer be used, might disappear in the next minor release.

Experimental

Intended for new, experimental features where we are looking for feedback. Use with caution, might be promoted to Maintained or Stable in the future, but might also be removed without prior notice.

Maintained

Intended for features that will not be changed in a backwards-incompatible way for at least the next minor release of the current major version. If scheduled for removal, it will be demoted to Deprecated first.

Stable

Intended for features that will not be changed in a backwards-incompatible way in the current major version.

Publicly visible classes will be annotated with @API(usage) where usage is one of the values in the above list, e.g. @API(Stable). This plan is expected to provide API callers a better perception of what they’re getting into, and the JUnit team the freedom to mercilessly change or remove unsupported APIs.

Open Test Alliance

As we have seen, the JUnit 5 architecture enables IDEs and build tools to use it as a facade for other testing frameworks (assuming those provide corresponding engines). With this approach tools can uniformly discover, execute, and assess tests without having to implement framework-specific support.

Or can they?

Test failures are typically expressed with exceptions. Unfortunately different test frameworks and assertion libraries do not generally use the same classes but rather implement their own variants (usually extending AssertionError or RuntimeException). This makes interoperability more complex than necessary and prevents uniform handling by tools.

To solve this problem the JUnit Lambda team split off a separate project, the Open Test Alliance for the JVM. This is their proposal:

Based on recent discussions with IDE and build tool developers from Eclipse, Gradle, and IntelliJ, the JUnit Lambda team is working on a proposal for an open source project to provide a minimal common foundation for testing libraries on the JVM.

The primary goal of the project is to enable testing frameworks like JUnit, TestNG, Spock, etc. and third-party assertion libraries like Hamcrest, AssertJ, etc. to use a common set of exceptions that IDEs and build tools can support in a consistent manner across all testing scenarios – for example, for consistent handling of failed assertions and failed assumptions as well as visualization of test execution in IDEs and reports.

Until now responses from those projects were mostly lacking. If you, dear reader, think this is a good idea, we encourage you to point out the Open Test Alliance to the maintainers of your frameworks of choice.

Compatibility

Given that JUnit can run test engines for versions 4 and 5 at the same time, it looks like a project can contain tests in both versions. And indeed, JUnit 5 occupies new namespaces: org.junit.jupiter, org.junit.platform, and org.junit.vintage. This means that there will be no conflicts when different JUnit versions are used in parallel and this allows a slow migration to JUnit 5.

Test libraries like Hamcrest and AssertJ, which communicate with JUnit via exceptions, will continue to work in the new version.

Summary

And with this we conclude part one of our JUnit 5 test drive. We’ve set up an environment to write and run tests and have seen how the API surface underwent an incremental evolution. With this you can start experimenting!

We also discussed how our tools were bound to JUnit 4 implementation details to such an extent that a new start was necessary. This was not the only reason, though. JUnit 4’s unsatisfactory extension model and the desire to use lambda expressions to define tests also lead to the rewrite.

The new architecture aims to avoid the errors of the past. It’s split into JUnit Jupiter, the library we use to write tests, and JUnit Platform, the platform tools can be built against, clearly separates these two concerns. And it also opens up the success of “JUnit the platform” to other testing frameworks that can now integrate with it.

In part two, we will take a closer look at how JUnit 5 tests can be run in IDEs, build tools, and even from the console. And finally, we will we see a couple of cool new features the new version brings to the table. The extension model is something to really look forward to...

About the Author

Nicolai Parlog is a software developer, author, and Java enthusiast. He constantly reads, thinks and writes about Java, and codes for a living as well as for fun. He's the editor of SitePoint's Java channel, writes a book about Project Jigsaw, and blogs about software development on CodeFX. You can follow Nicolai on Twitter.