Loading...

Integrate tests with the Allure report tool

By  Grzegorz Szczutkowski | December 11, 2020 | Selenium WebDriver

 Many times I was asked how to create reports from automated tests, what tools use for that and how to connect those tests to manual test cases which are kept in any test management tool. I know that there are many ways of achieving that. In this article I will show you based on the Allure Report tool how to generate nice and descriptive report, which is clear for management and contains all important data like steps, screenshots, links to test management tool and bugs tracing tool. All that is automatically generated and quite easy to achieve.

For article purposes I have created simple end-to-end test framework which, hmm... do nothing but gives us a structure of testing framework which we want to integrate with the Allure Report tool. Based on this framework I will explain most important features of the tool and illustrate how to use them.
Our one and only page is shown below: 

public class GooglePage {
    private static final Logger LOGGER = LoggerFactory.getLogger(GooglePage.class);
    private String searchCriteria = "";

    public GooglePage clickOnInputField() {
        LOGGER.info("Clicking on google search field.");
        return this;
    }

    public GooglePage enterTextInSearchField(String text) {
        LOGGER.info("Entering text '{}' in the google input field.", text);
        searchCriteria = text;
        return this;
    }

    public GooglePage clickOnImFeelingLucky() {
        LOGGER.info("Clicking on I'm feeling lucky button.");
        return this;
    }

    public GooglePage clickOnSearchButton() {
        LOGGER.info("Clicking on google search field.");
        return this;
    }

    public String firstItemTitle() {
        LOGGER.info("Clicking on google search field.");
        if (searchCriteria.equals("Biggest country in the world")) {
            return "Russia";
        } else {
            return "";
        }
    }
}


As we can see on the listing above in the GooglePage we have a few methods which are "responsible" for performing actions on google page. We have also a class with tests which uses the GooglePage object. Now we can say that our test framework is ready. The listing of this test class is shown below: 

public class GoogleSearchTest {
    private static final String GOOGLE_URL = "https://google.com";

    @Test
    public void searchForLongCriteriaShouldMatchResults() {
        GooglePage resultsPage = openPage()
                .clickOnInputField()
                .enterTextInSearchField("Very long searching criteria")
                .clickOnSearchButton();

        assertThat(resultsPage.firstItemTitle()).isEqualTo("");
    }

    @Test
    public void searchShouldNotBePerformedWhenNoCriteria() {
        GooglePage resultsPage = openPage(GOOGLE_URL)
                .clickOnInputField()
                .enterTextInSearchField("")
                .clickOnSearchButton();

        assertThat(resultsPage.firstItemTitle()).isEqualTo("");
    }

    @Test
    public void itShouldBePossibleToSearchPhrase() {
        GooglePage resultsPage = openPage(GOOGLE_URL)
                .clickOnInputField()
                .enterTextInSearchField("Biggest country in the world")
                .clickOnSearchButton();

        assertThat(resultsPage.firstItemTitle()).isEqualTo("Russia");
    }

    @Test
    public void itShouldBePossibleToSearchSingleWord() {
        GooglePage resultsPage = openPage(GOOGLE_URL)
                .clickOnInputField()
                .enterTextInSearchField("Yacht")
                .clickOnSearchButton();

        assertThat(resultsPage.firstItemTitle()).isEqualTo("Christina O");
    }

    @Test
    public void iAmFeelingLuckyShouldBeBlockedForSecondClick() {

        GooglePage resultsPage = openPage(GOOGLE_URL)
                .clickOnInputField()
                .clickOnImFeelingLucky();

    }

    @Test
    public void iAmFeelingLuckyShouldReturnDesiredResults() {

        GooglePage resultsPage = openPage(GOOGLE_URL)
                .clickOnInputField()
                .clickOnImFeelingLucky();

        assertThat(resultsPage.firstItemTitle()).isEqualTo("Christina O");
    }

    @After
    public void tearDown() throws IOException, URISyntaxException {
        takeScreenshot();
    }

    private GooglePage openPage(String... url) {
        if (url[0].equals("https://google.com")) {
            return new GooglePage();
        } else {
            return null;
        }
    }
}


When we have our test framework ready we want to integrate it with Allure. First step for doing that is adding dependency: 

<dependency>
	<groupId>io.qameta.allure</groupId>
	<artifactId>allure-junit4</artifactId>
	<version>2.13.1</version>
</dependency>


and two sections with plugins to our pom.xml 

<build>
	<plugins>
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-surefire-plugin</artifactId>
			<version>2.22.2</version>
			<configuration>
				<useSystemClassLoader>false</useSystemClassLoader>
				<argLine>
					-javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
				</argLine>
				<properties>
					<property>
						<name>listener</name>
						<value>io.qameta.allure.junit4.AllureJunit4</value>
					</property>
				</properties>
				<systemProperties>
					<property>
						<name>allure.results.directory</name>
						<value>${project.build.directory}/allure-results</value>
					</property>
				</systemProperties>
			</configuration>
			<dependencies>
				<dependency>
					<groupId>org.aspectj</groupId>
					<artifactId>aspectjweaver</artifactId>
					<version>${aspectj.version}</version>
				</dependency>
			</dependencies>
		</plugin>
		<plugin>
			<groupId>io.qameta.allure</groupId>
			<artifactId>allure-maven</artifactId>
			<version>2.10.0</version>
		</plugin>
	</plugins>
</build>


The first plugin is responsible for generating results readable for the Allure Report tool. The second one helps us generate and even deploy the report in html format. When the pom.xml is ready we can move to decorate our code with annotations which will be responsible for putting concrete data to the test report. Let's start from adding @Step annotations to our method in page object: 

    @Step("Click on google search field.")
    public GooglePage clickOnInputField() {
        LOGGER.info("Clicking on google search field.");
        return this;
    }

    @Step("Enter {text} in google search field.")
    public GooglePage enterTextInSearchField(String text) {
        LOGGER.info("Entering text '{}' in the google input field.", text);
        searchCriteria = text;
        return this;
    }

    @Step("Click [I'm feeling lucky] button.")
    public GooglePage clickOnImFeelingLucky() {
        LOGGER.info("Clicking on I'm feeling lucky button.");
        return this;
    }

    @Step("Click on [Search] button.")
    public GooglePage clickOnSearchButton() {
        LOGGER.info("Clicking on google search field.");
        return this;
    }


Whenever the method with step annotation is executed, then the text from the Step is added as a step in the test case in the test report. The steps in the report are displayed in the same order as methods were executed. In the steps we can also use variables, e.g. {text} part will be filled in on the report by value which was passed to the method to text argument. Allure gives us much more possibilities to describe the tests and pass the data to the report. Most of them are illustrated in the updated test class: 

@Epic("TCE-001: Support search engine")
public class GoogleSearchTest {
    private static final String GOOGLE_URL = "https://google.com";

    @BeforeClass
    public static void setUp() {
        config().initializeAllureSettings();
    }

    @TmsLink("TestCase-001")
    @Story("TCS-003: Support searching by text criteria.")
    @DisplayName("Search engine should work with long search criteria")
    @Test
    public void searchForLongCriteriaShouldMatchResults() {
        GooglePage resultsPage = openPage()
                .clickOnInputField()
                .enterTextInSearchField("Very long searching criteria")
                .clickOnSearchButton();

        assertThat(resultsPage.firstItemTitle()).isEqualTo("");
    }

    @TmsLink("TestCase-002")
    @Story("TCS-003: Support searching by text criteria.")
    @DisplayName("Search should not be possible when there is no search criteria")
    @Severity(SeverityLevel.MINOR)
    @Test
    public void searchShouldNotBePerformedWhenNoCriteria() {
        GooglePage resultsPage = openPage(GOOGLE_URL)
                .clickOnInputField()
                .enterTextInSearchField("")
                .clickOnSearchButton();

        assertThat(resultsPage.firstItemTitle()).isEqualTo("");
    }

    @TmsLink("TestCase-002")
    @Story("TCS-003: Support searching by text criteria.")
    @Severity(SeverityLevel.NORMAL)
    @Test
    public void itShouldBePossibleToSearchPhrase() {
        GooglePage resultsPage = openPage(GOOGLE_URL)
                .clickOnInputField()
                .enterTextInSearchField("Biggest country in the world")
                .clickOnSearchButton();

        assertThat(resultsPage.firstItemTitle()).isEqualTo("Russia");
    }

    @TmsLink("TestCase-003")
    @Issue("438")
    @Story("TCS-003: Support searching by text criteria.")
    @DisplayName("Search engine should work with single word")
    @Description("Search engine should return pages which contain in the title or content desired word."
            + " Order of the returned pages should be ascending by popularity.")
    @Severity(SeverityLevel.CRITICAL)
    @Test
    public void itShouldBePossibleToSearchSingleWord() {
        GooglePage resultsPage = openPage(GOOGLE_URL)
                .clickOnInputField()
                .enterTextInSearchField("Yacht")
                .clickOnSearchButton();

        assertThat(resultsPage.firstItemTitle()).isEqualTo("Christina O");
    }

    @Ignore("Automation test is not finished")
    @Stories({
            @Story("TCS-002: Create search engine for lucky searching"),
            @Story("TCS-004: Support searching lucky results.")
    })
    @Severity(SeverityLevel.MINOR)
    @Test
    public void iAmFeelingLuckyShouldBeBlockedForSecondClick() {
        GooglePage resultsPage = openPage(GOOGLE_URL)
                .clickOnInputField()
                .clickOnImFeelingLucky();

        assertThat(resultsPage.clickOnImFeelingLucky()).isEqualTo(resultsPage);
    }

    @Stories({
            @Story("TCS-002: Create search engine for lucky searching"),
            @Story("TCS-004: Support searching lucky results.")
    })
    @Severity(SeverityLevel.CRITICAL)
    @Test
    public void iAmFeelingLuckyShouldReturnDesiredResults() {

        GooglePage resultsPage = openPage(GOOGLE_URL)
                .clickOnInputField()
                .clickOnImFeelingLucky();

        assertThat(resultsPage.firstItemTitle()).isEqualTo("Christina");
    }

    @After
    public void tearDown() throws IOException, URISyntaxException {
        savePngAttachment(takeScreenshot());
    }

    @Step("Open browser and navigate to {url}")
    private GooglePage openPage(String... url) {
        if (url[0].equals("https://google.com")) {
            return new GooglePage();
        } else {
            return null;
        }
    }

    @Attachment(value = "Screenshot from web page.", type = "image/png")
    public byte[] savePngAttachment(URL screenshotUrl) throws URISyntaxException, IOException {
        if (screenshotUrl == null) {
            fail("Couldn't find resource 'screenshot.png'");
        }
        return Files.readAllBytes(Paths.get(screenshotUrl.toURI()));
    }

    public URL takeScreenshot(){
        return getClass().getClassLoader().getResource("screenshot.png");
    }
}


I have added the @Epic annotation so all test cases in the class cover different functionalities from this very epic. Annotation @Story marks the test case that it covers the particular story. We can use @Stories annotation when we want to mark that one test case covers more than one story. @DisplayName and @Description are the annotations which contain the name of the test case visible in the report and the additional description if you want to add it to the test case.
There are also two useful annotations which connect our report with our test management tool and bug tracking tool. Annotations can be configured by e.g. passing parametrized link to the tool via system properties. If for example, your bug tracing tool is github then you have to set system property e.g. to allure.link.issue.pattern=https://github.com/allure-framework/allure-java/issues/{} then when you have any test case annotated with @Issue("492") then from the test report and results of this particular test case you can navigate to the bug under the link https://github.com/allure-framework/allure-java/issues/492. The same is with @TmsLink annotation the only difference is that you have to set the test management tool link pattern to allure.link.tms.pattern system property.
The last worth mentioning thing is @Attachment annotation which allows us to pass any external file to the test report (e.g. screenshots, log files). The only thing you have to remember is that the method have to return the file as the bytes table.


The easiest way to learn how to use this tool is to play with it. You can start with example from this article which is available here.

Categories
Tags