Loading...

Preventing flaky tests in your automation suite

By Grzegorz Szczutkowski | August 17, 2017 | Selenium WebDriver

 There are many ways of dealing with unstable tests. Some of them are very complex and can be called more like a process but some are are very simple to introduce. In this article I will focus on one very simple and chip idea to introduce - mainly if you using java and junit. The idea is in general to run newly created test several times (e.g. 50 times) before merging it to branch with tests run regularly.

To achieve that you need two things. Thirst of you have to create test category for marking test cases which need to be verified from stability point of view. I have described the way of creating test categories it in post How to categorise automation tests. Thus, create new empty interface e.g. StabilityCheckCategory and with use of this category create new maven profile like shown below:

<profile>
	<id>stability-check</id>
	<properties>
		<test.case.excluded.groups/>
		<test.case.groups>com.testcraftsmanship.model.configuration.StabilityCheckCategory</test.case.groups>
	</properties>
</profile>


The next thing to add is a new implementation of TesRule in which you will check if test is marked with StabilityCheckCategory annotation. If yes, then it should run the statement in the loop desired number of times (number of times kept in the variable Constants.NUMBER_OF_REPETITIONS_IN_STABILITY_CHECK). If the test case is not marked with the StabilityCheckCategory annotation, then test will be run with use of default Statement implementation. Below is the example implementation of that TestRule:

package com.testcraftsmanship.model.base;

import com.testcraftsmanship.model.configuration.Constants;
import com.testcraftsmanship.model.configuration.StabilityCheckCategory;
import org.junit.experimental.categories.Category;
import org.junit.rules.TestRule;
import org.junit.runner.Description;
import org.junit.runners.model.Statement;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.Arrays;
import java.util.Optional;

import static com.testcraftsmanship.model.base.DescriptionDataExtractor.getCategory;
import static com.testcraftsmanship.model.base.DescriptionDataExtractor.getClassName;
import static com.testcraftsmanship.model.base.DescriptionDataExtractor.getMethodName;

public class RepeatRule implements TestRule {
    private static final Logger LOGGER = LoggerFactory.getLogger(RepeatRule.class);

    private static class RepeatStatement extends Statement {
        private final Statement statement;
        private final int repeat;

        RepeatStatement(Statement statement, int repeat) {
            this.statement = statement;
            this.repeat = repeat;
        }

        @Override
        public void evaluate() throws Throwable {
            for (int iteration = 0; iteration < repeat; iteration++) {
                LOGGER.info("Test case run number {} of {}", iteration + 1, repeat);
                statement.evaluate();
            }
        }
    }

    @Override
    public Statement apply(Statement statement, Description description) {
        Statement result = statement;
        Optional<Category> category = getCategory(description);
        if (isStabilityCheckCategory(category)) {
            LOGGER.info("Test case {} from class {} will be repeated {} times",
                    getMethodName(description),
                    getClassName(description),
                    Constants.NUMBER_OF_REPETITIONS_IN_STABILITY_CHECK);
            result = new RepeatStatement(statement,
                    Constants.NUMBER_OF_REPETITIONS_IN_STABILITY_CHECK);
        }
        return result;
    }

    private boolean isStabilityCheckCategory(Optional<Category> category) {
        return category.isPresent() &&
                Arrays.asList(category.get().value()).contains(StabilityCheckCategory.class);
    }
}


In the class above I have moved some functionality to the external class to make code more clean. Class DescriptionDataExtractor contains only methods which extracts class name with test case, test method name and test category from description object. Below is implementation of this class:

package com.testcraftsmanship.model.base;

import org.junit.experimental.categories.Category;
import org.junit.runner.Description;

import java.util.Optional;

class DescriptionDataExtractor {
    static String getClassName(Description description) {
        return description.getTestClass()
                .getName().split("\\.")[description
                    .getTestClass().getName().split("\\.").length - 1];
    }

    static String getMethodName(Description description) {
        return description.getMethodName().split("\\[")[0];
    }

    static Optional<Category> getCategory(Description description) {
        return Optional.ofNullable(description.getAnnotation(Category.class));
    }
}


The last thing you have to do is adding the newly created rule to our test classes: 

@Rule
public RepeatRule repeatRule = new RepeatRule();


I suggest to create an abstract class e.g. BaseTest in which this Rule and all other general test related things can be added. All your test classes should extend this class and by that the rule is applied to all your test cases.

Categories
Tags