Executing parallel test cases in multiple agents using Azure Pipelines and Provar Test Plans

By Mohit Gupta, Technical Sales Manager at Provar

Do you run hundreds (or maybe even a thousand or more) test cases as part of your regression testing each day? Do you also need to run tests simulating a variety of different browsers and in various environments? If so, you are likely familiar with the challenges associated with creating, managing and maintaining an incredibly vast, dynamic and complex scope of testing. 

If this sounds familiar, you might consider using Provar Test Plans in conjunction with Azure Pipelines to simplify your workload. Let’s take a closer look. 

Provar Test Plans

Provar Test Plans are designed to help users streamline the testing processes associated with supporting new software releases in a structured and easily repeatable manner. Test plans give you the tools you need to specify which (and how) test cases need to be run to test a new release or milestone. 

You can build multiple test plans and run a repeatable collection of tests per each release cycle and make global changes to the environment settings such as browser settings, the build number and the build server. Plus, you can get consolidated reports of your results.

A real-world example and use case:

• You need to regularly run 500 test cases 
• The average test case takes two minutes for the execution
• Execution needs to be completed on both Chrome and Firefox browsers

Given that, it would take roughly 2,000 minutes (~33.3 hours)  to run those tests sequentially. On top of that, if you need to run those tests several times per week, your workload just got exponentially more difficult.  

Note: While you can use Provar to execute test cases in parallel via ANT (either on the same agent to get a single test report or via multiple agents with separate test reports) this could impact memory consumption during execution.

The solution

Azure Devops has an easy solution for the problem statement we mentioned above. Using Azure Pipelines, it allows us to execute a set of different test plans on different agents in parallel.

Let’s continue with the above case study. We will divide 500 test cases in a set of two test plans with 250 test cases in each and both need to be executed for Chrome and Firefox browsers. With Azure Pipelines, we can execute these tests on four different agents in parallel, which would be as follows:

• Agent 1: Test Plan 1 with Chrome
• Agent 2: Test Plan 2 with Chrome
• Agent 3: Test Plan 1 with Firefox
• Agent 4: Test Plan 2 with Firefox

In this scenario, you can accelerate test execution. Each agent would take only 500 minutes (8.3 Hrs) and with a lower impact on system performance.

How to do it

To achieve this, we will use the multi-configuration feature of Azure Pipelines. This allows us to pass multiple parameters to the pipeline and use them to control the execution on different agents. You can find this option under Pipeline > Agent. 

Here are a few important things to consider: 

Multiplier: This is the parameter under Variables which we will define in a moment.

Maximum number of agents: You can specify the count agents on which execution needs to be done in parallel. 

Step 1: Define the configuration in Variables.

Step 2: Provide the multiplier for each agent.

Go to the agent and provide the variables created under the Multipliers section. 

Step 3: Provide the configuration details under the ANT task.

Step 4:  Do the required change in the build.xml files.

And you are done! Just execute Azure Pipelines and you will see the magic happen.

For more information, you can contact us at [email protected].