How to Integrate Selenium and TestRail

What is Selenium?

Selenium is an open-source tool used to automate web-based applications on various browsers and environments.

What is TestRail?

TestRail is extensive web-based test case management software used to effectively manage, track, construct and organize your software testing efforts.

Integration between Selenium and TestRail defines the result of a unique linked Test ID in TestRail and depends on the result a user is getting from the Selenium automation suite.

The following are the two ways in which you can update your TestRail results after a test case execution:

• Create a TestNG listener that will listen for test results and call the TestRail API to update it with any results it finds
• Call the TestRail API inside the test method to update results

Two methods to update test results are:

• add_result
• add_result_for_case

The difference between the two is that while “add_result” takes the Test ID, “add_result_for_case” takes the Case ID and Run ID. We can use any method depending on the automation approach chosen.

An advantage of using “add_result_for_case” over “add_result” is that while creating a test run as per the iteration, the test ID changes corresponding to the test run ID, but our test case ID remains the same. This makes it easy to maintain the test case ID and run the automation suite.

Why an integration between TestRail and Selenium is required:

In a real-time project, automation runs one iteration behind schedule. Therefore, it is necessary to add around 10% of manual test cases in each iteration (which will be automated in next iteration).

Let’s take an example of an iteration (sprint32):

Total number of test cases in TestRail = 1000

Total number of test cases added in sprint32 = 100          <== Manual Test Case

Total number of automated test cases in TestRail = 900          <== Automated Test Case

During regression testing of sprint32, the regression automation suite has both manual and automated test cases. By integrating TestRail and Selenium, we can easily update the status of the automated test case in TestRail with detailed failure messages and screenshots. The testing team can then update the remaining 10% manual test cases. This approach saves both time and the manual effort required to update the status in TestRail.

Advantages of TestRail and Selenium Integration:

• Saves time and manual effort while updating the test case status
• Provides an estimate of how many manual test cases have been automated and integrated with TestRail so they can be marked as automated for future reference
• Gives a real picture of the release before starting manual regression testing
• Helps in streamlining the testing process
• Helps in verifying RTM (Requirement Traceability Matrix) in JIRA

Sample Test Code :

package test;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import org.json.JSONObject;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.testng.Assert;
import org.testng.annotations.Test;
import util.APIClient;
import util.APIException;
public class testgoogle {

    public static String TEST_RUN_ID                = <Test RUN ID>;
    public static String TESTRAIL_USERNAME          = <Test Rail Username>;
    public static String TESTRAIL_PASSWORD          = <Test Rail Password>;
    public static String RAILS_ENGINE_URL           = <Test Rail URL>;
    public static final int TEST_CASE_PASSED_STATUS = 1;
    public static final int TEST_CASE_FAILED_STATUS = 5;
    WebDriver driver;
    
    @Test(description = "Verify that Title appearing on the webpage is as expected. TC_ID=C120254")
    public void verifyTitle() throws Exception, APIException {
        driver= new FirefoxDriver();
        driver.get("https://www.google.co.in");
        String actualTilte = driver.getTitle();
        if (actualTilte.contains("Google")) {
            Assert.assertTrue(actualTilte.contains("Google"));
            testgoogle.addResultForTestCase("120254", TEST_CASE_PASSED_STATUS, "");
        }
else{
		testgoogle.addResultForTestCase("120254", TEST_CASE_FAILED_STATUS, "");
	}

        driver.quit();
    }


    public static void addResultForTestCase(String testCaseId, int status,
            String error) throws IOException, APIException {

        String testRunId = TEST_RUN_ID;

        APIClient client = new APIClient(RAILS_ENGINE_URL);
        client.setUser(TESTRAIL_USERNAME);
        client.setPassword(TESTRAIL_PASSWORD);
        Map data = new HashMap();
        data.put("status_id", status);
        data.put("comment", "Test Executed - Status updated automatically from Selenium test automation.");
        client.sendPost("add_result_for_case/"+testRunId+"/"+testCaseId+"",data );

    }

}

Screenshot of TestRail before execution of automation suite:

Code Snippet:

Screenshot of TestRail after execution of automation suite:

Sample API methods :

There is a Java API binding available at: https://github.com/gurock/testrail-api

Documentation for the API is available here.

You instantiate the API connection in Java with:

import com.gurock.testrail.APIClient;
import com.gurock.testrail.APIException;
import java.util.Map;
import java.util.HashMap;
import org.json.simple.JSONObject;

public class Program
{
    public static void main(String[] args) throws Exception
    {
        APIClient client = new APIClient("http://<server>/testrail/");
        client.setUser("..");
        client.setPassword("..");
    }
}

Here’s an example of a GET request:

APIClient client = new APIClient("http://<server>/testrail/");
client.setUser("..");
client.setPassword("..");
JSONObject c = (JSONObject) client.sendGet("get_case/1");
System.out.println(c.get("title"));

And here’s a POST request:

Map data = new HashMap();
data.put("status_id", new Integer(1));
data.put("comment", "This test worked fine!");
JSONObject r = (JSONObject) client.sendPost("add_result_for_case/1/1", data);