Executing Parallel Test Cases in Multiple Agents using Azure Pipelines and Provar Test Plans

By Mohit Gupta, Technical Sales Manager at Provar

Do you run hundreds (or maybe even a thousand or more) of test cases as part of your regression testing each day? Do you also need to run tests simulating various browsers and in multiple environments? If so, you are likely familiar with the challenges associated with creating, managing, and maintaining an incredibly vast, dynamic, and complex scope of testing with Azure Pipelines.

If this sounds familiar, you might consider using Provar Test Plans with Azure Pipelines to simplify your workload. Let’s take a closer look.

Provar Test Plans

Provar Test Plans are designed to help users streamline the testing processes associated with supporting new software releases in a structured and easily repeatable manner. Test plans give you the tools to specify which (and how) test cases need to be run to test a new release or milestone.

You can build multiple test plans, run a repeatable collection of tests per each release cycle, and make global changes to the environment settings, such as browser settings, the build number, and the build server. Plus, you can get consolidated reports of your results.

A real-world example and use case:

You need to run 500 test cases regularly
The average test case takes two minutes for the execution
Execution needs to be completed on both Chrome and Firefox browsers

Running those tests sequentially would take roughly 2,000 minutes (~33.3 hours). On top of that, if you need to run those tests several times per week, your workload gets exponentially more difficult.

Note: While you can use Provar to execute test cases in parallel via ANT (either on the same agent to get a single test report or via multiple agents with separate test reports), this could impact memory consumption during execution.

The Solution

Azure DevOps has an easy solution for the problem statement we mentioned above. Azure Pipelines allows us to execute different parallel test plans on other agents.

Let’s continue with the above case study. We will divide 500 test cases into two test plans with 250 test cases in each, and both need to be executed for Chrome and Firefox browsers. With Azure Pipelines, we can perform these tests on four different agents in parallel, which would be as follows:

Agent 1: Test Plan 1 with Chrome
Agent 2: Test Plan 2 with Chrome
Agent 3: Test Plan 1 with Firefox
Agent 4: Test Plan 2 with Firefox

In this scenario, you can accelerate test execution. Each agent would take only 500 minutes (8.3 Hrs), with a lower impact on system performance.

How To Do It

To achieve this, we will use the multi-configuration feature of Azure Pipelines. This allows us to pass multiple parameters to the pipeline and use them to control the execution of different agents. You can find this option under Pipeline > Agent.

Here are a few essential things to consider:

Multiplier: This is the parameter under Variables we will define briefly.

The maximum number of agents: You can specify the number of agents to execute the task in parallel.