How to categorise automation tests

 Many times in my career I was obligated to divide automation tests into suites. Dividing helps in many situations like moving our flaky tests into separate run or having smoke tests which won't take ours but still check major functions of tested applications in short time. In this post I refer to automation tests written in Java plus Selenium WebDriver and show how to easily create and use such categories with junit and maven.

First of all we have to create an empty classes which describes all our suites (e.g. UnstableTestCategory.java, SmokeTestCategory.java). Note that not all test cases have to be marked - we will be able to select test cases by e.g. excluding suite from all test cases.

When classes with test suites are created we can create maven profiles in which we indicate which suites will be run. I have added properties to POM file with default values. In testcase.groups property I have locations of all categories which will be run (if property is empty then all test cases will be run). In testcase.excluded.groups property I have locations of all categories which will be excluded from the run. If you need more then one location then you can separate them by the comma.

<properties>
   <testcase.groups/>
   <testcase.excluded.groups>
      com.testcraftsmanship.framework.categories.UnstableTestsCategory
   </testcase.excluded.groups>
</properties>

Thus when I run tests from maven with no profile then all my test cases excluded UnstableTestsCategory will be run. I have added two example profiles to the POM file which overwrite default values of the properties.

<profiles>
   <profile>
      <id>unstable-tests</id>
      <properties>
         <testcase.excluded.groups/>
         <testcase.groups>
            com.testcraftsmanship.framework.categories.UnstableTestsCategory
         </testcase.groups>
      </properties>
   </profile>
   <profile>
      <id>smoke-tests</id>
      <properties>
         <testcase.excluded.groups/>
         <testcase.groups>
            com.testcraftsmanship.framework.categories.SmokeTestCategory
         </testcase.groups>
      </properties>
   </profile>
</profiles>

Thus when I run tests with unstable-tests profile then only UnstableTestsCategory will be run and when I run them with profile smoke-tests then only SmokeTestCategory will be run.

To make it working we have to add maven-surefire-plugin in which in configuration we are assigning values from previously defined properties.

<plugins>
   <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>${maven.surefire.version}</version>
      <dependencies>
         <dependency>
            <groupId>org.apache.maven.surefire</groupId>
            <artifactId>surefire-junit47</artifactId>
            <version>${maven.surefire.version}</version>
         </dependency>
      </dependencies>
      <configuration>
         <groups>${test.case.groups}</groups>
         <excludedGroups>${test.case.excluded.groups}</excludedGroups>
      </configuration>
   </plugin>
</plugins>

Thats all what we have to add on POM file side. Now we have to assigned categories to our tests and we can simply do it by adding @Category annotation to the test like shown on example below:

@Category(SmokeTestCategory.class)
@Test
public void userDataShouldNotBeOpenedWhenHaveNoReadPermission() {
   ...
}

When we assign all categoies to desired tests we can simply run the suite by running mvn test -P{profile_name}.