Junit Deepdive Part2

This time i want to cover the topic of aggregating JUnit tests which is also part of my blog series Is your JUnit knowhow up to date? The last article can be found here. So let us get started.

Suites

A JUnit Suite is the simplest and oldest way to aggregate JUnit tests. You can use it by annotating your test suite with @Suite as you can see in the following code snippet:

1 @RunWith(Suite.class)
2 @Suite.SuiteClasses({CalculatorSimpleTest.class, CalculatorPerformanceTest.class})
3 public class CalculatorSuite {
4 }

Basically you tell JUnit to look out for the class array of the @Suite.SuiteClasses which contains all the classes of the suite. But there is a little bit more. Suite itself inherits from ParentRunner which i covered in the last article. Therefore Suites have a lifecycle but in comparison to the default runner only include the Class Ready lifecycle methods which are:

But although it is technically possible to used them it does not mean that it is good. In general a suite class should only act as an aggregation container. One reason for that is that suite classes can be listed in suites and categories itself so that you can build up hierarchical test structures. If you now use lifecycle methods in your suites you probably decrease understandability and your isolation between your suites.

I also took a deep look at the Suite.SuiteClasses annotation which i considered at first as a bad solution to the problem of aggregating test classes and suites. The reason for that was the declarative nature of it which means that you have to declare every single class or suite. Even with IDE support nowadays this could lead to some work if you have many classes. Considering bug fixes or enhancements of a system there is also the problem that those new tests maybe do not make it in the test suites and therefore could lead to bad reports about the health or quality of the aggregation part.
On the second look i changed my mind. The first reason for that was that there are already some libraries with whom you can overcome the declarative nature problem. The two most promising ones should be:

The second reason came after thinking about when aggregate tests anyway which i would do for the following topics:
There are probably more topics by which you can aggregate your tests. The point is that in such cases you explicitly want to declare which tests are in that suite and which are not.

FYI: In the old days of JUnit 3 suites were recognized by a public static Test suite() method in which you had to add the test classes.

Categories

A more flexible way to aggregate tests are Categories which were introduced with JUnit 4.8 and strangely enough are still in the experimental package. The basic concept is similar to suites which means that there is a Categories class which is a JUnit runner (in detail inherits from Suite) and has to be declared with @RunWith(Categories.class) on top of your aggregation container. You also have to declare your test classes with @Suite.SuiteClasses.
The new part is that you can mark your test classes and/or test methods with @Category and a marker class which is used as a filter in your test aggregation container. For that there are two more annotations @Categories.IncludeCategory and @Categories.ExcludeCategory which take a category filter class (the default behavior, without any include or exclude, includes all test methods). The only limitation i found so far is that you can use only one category marker class on the test aggregation container and that you can not repeat those annotations.
An example of an aggregate container would be look like this:

1 @RunWith(Categories.class)
2 @Categories.IncludeCategory(SlowTests.class)
3 @Suite.SuiteClasses( { CalculatorSimpleTest.class, CalculatorPerformanceTest.class})
4 public class OnlySlowTestSuite {}
5 
6 public interface SlowTests{}

In this example i had declared a category suite which uses the test classes CalculatorSimpleTest and CalculatorPerformanceTest and include from that classes only the test methods which are annotated with @Category(SlowTests.class). A @Category annotation can be used on class and/or on method level and expects an array of classes. In general it is recommended to use only one class because otherwise it could be really difficult to understand your suite filters considering including and excluding.
In the example above i declare the SlowTests category class marker.

 1 public class CalculatorSimpleTest {
 2 
 3     //snip..
 4 
 5     @Category(SlowTests.class)
 6     @Test
 7     public void testSubstract() {
 8         for (int i = 0; i < 10000; i++) {
 9             try {Thread.sleep(10l);} catch (InterruptedException e) {
10                 e.printStackTrace();
11             }
12 
13             String errorMessage = "Substracting failed";
14             int expected = 1;
15             int add = calculator.substract(i + 1, i);
16             assertEquals(errorMessage, expected, add);
17         }
18     }
19 }

If you want to skip all slowtests you would exchange IncludeCategory with ExcludeCategory.

At the time of writing this article JUnit 4.12 beta3 is released which already denotes a little change in the lifecycle handling considering categories. Until now it was possible to use @Category on lifecycle methods like @Before which made those tests very hard to understand. Therefore JUnit introduced a validation handling which prohibits this usage.