By Mohit Gupta, Technical Sales Manager at Provar
Do you run hundreds (or maybe even a thousand or more) of test cases as part of your regression testing each day? Do you also need to run tests simulating a variety of different browsers and in various environments? If so, you are likely familiar with the challenges associated with creating, managing, and maintaining an incredibly vast, dynamic, and complex scope of testing with Azure Pipelines.
If this sounds familiar, you might consider using Provar Test Plans in conjunction with Azure Pipelines to simplify your workload. Let’s take a closer look.
Provar Test Plans
Provar Test Plans are designed to help users streamline the testing processes associated with supporting new software releases in a structured and easily repeatable manner. Test plans give you the tools you need to specify which (and how) test cases need to be run to test a new release or milestone.
You can build multiple test plans and run a repeatable collection of tests per each release cycle and make global changes to the environment settings such as browser settings, the build number, and the build server. Plus, you can get consolidated reports of your results.
A real-world example and use case:
- You need to regularly run 500 test cases
- The average test case takes two minutes for the execution
- Execution needs to be completed on both Chrome and Firefox browsers
Given that, it would take roughly 2,000 minutes (~33.3 hours) to run those tests sequentially. On top of that, if you need to run those tests several times per week, your workload just got exponentially more difficult.
Note: While you can use Provar to execute test cases in parallel via ANT (either on the same agent to get a single test report or via multiple agents with separate test reports) this could impact memory consumption during execution.
The Solution
Azure DevOps has an easy solution for the problem statement we mentioned above. Using Azure Pipelines, allows us to execute a set of different test plans on different agents in parallel.
Let’s continue with the above case study. We will divide 500 test cases into a set of two test plans with 250 test cases in each and both need to be executed for Chrome and Firefox browsers. With Azure Pipelines, we can execute these tests on four different agents in parallel, which would be as follows:
- Agent 1: Test Plan 1 with Chrome
- Agent 2: Test Plan 2 with Chrome
- Agent 3: Test Plan 1 with Firefox
- Agent 4: Test Plan 2 with Firefox
In this scenario, you can accelerate test execution. Each agent would take only 500 minutes (8.3 Hrs) and with a lower impact on system performance.
How To Do It
To achieve this, we will use the multi-configuration feature of Azure Pipelines. This allows us to pass multiple parameters to the pipeline and use them to control the execution of different agents. You can find this option under Pipeline > Agent.
Here are a few important things to consider:
Multiplier: This is the parameter under Variables which we will define in a moment.
Maximum number of agents: You can specify the count of agents on which execution needs to be done in parallel.
Step 1: Define the configuration in Variables.
Step 2: Provide the multiplier for each agent.
Go to the agent and provide the variables created under the Multipliers section.
Step 3: Provide the configuration details under the ANT task.
Step 4: Do the required change in the build.xml files.
And you are done! Just execute Azure Pipelines and you will see the magic happen.
For more information, you can contact us at [email protected].
More from our blog
You might also be interested in
