By Mohit Gupta, Technical Sales Manager at Provar
Do you run hundreds (or maybe even a thousand or more) of test cases as part of your regression testing each day? Do you also need to run tests simulating various browsers and in multiple environments? If so, you are likely familiar with the challenges associated with creating, managing, and maintaining an incredibly vast, dynamic, and complex scope of testing with Azure Pipelines.
If this sounds familiar, you might consider using Provar Test Plans with Azure Pipelines to simplify your workload. Let’s take a closer look.
Provar Test Plans
Provar Test Plans are designed to help users streamline the testing processes associated with supporting new software releases in a structured and easily repeatable manner. Test plans give you the tools to specify which (and how) test cases need to be run to test a new release or milestone.
You can build multiple test plans, run a repeatable collection of tests per each release cycle, and make global changes to the environment settings, such as browser settings, the build number, and the build server. Plus, you can get consolidated reports of your results.
A real-world example and use case:
- You need to run 500 test cases regularly
- The average test case takes two minutes for the execution
- Execution needs to be completed on both Chrome and Firefox browsers
Running those tests sequentially would take roughly 2,000 minutes (~33.3 hours). On top of that, if you need to run those tests several times per week, your workload gets exponentially more difficult.
Note: While you can use Provar to execute test cases in parallel via ANT (either on the same agent to get a single test report or via multiple agents with separate test reports), this could impact memory consumption during execution.
The Solution
Azure DevOps has an easy solution for the problem statement we mentioned above. Azure Pipelines allows us to execute different parallel test plans on other agents.
Let’s continue with the above case study. We will divide 500 test cases into two test plans with 250 test cases in each, and both need to be executed for Chrome and Firefox browsers. With Azure Pipelines, we can perform these tests on four different agents in parallel, which would be as follows:
- Agent 1: Test Plan 1 with Chrome
- Agent 2: Test Plan 2 with Chrome
- Agent 3: Test Plan 1 with Firefox
- Agent 4: Test Plan 2 with Firefox
In this scenario, you can accelerate test execution. Each agent would take only 500 minutes (8.3 Hrs), with a lower impact on system performance.
How To Do It
To achieve this, we will use the multi-configuration feature of Azure Pipelines. This allows us to pass multiple parameters to the pipeline and use them to control the execution of different agents. You can find this option under Pipeline > Agent.
Here are a few essential things to consider:
Multiplier: This is the parameter under Variables we will define briefly.
The maximum number of agents: You can specify the number of agents to execute the task in parallel.
Step 1: Define the configuration in Variables.
Step 2: Provide the multiplier for each agent.
Go to the agent and provide the variables created under the Multipliers section.
Step 3: Provide the configuration details under the ANT task.
Step 4: Make the required change in the build.xml files.
That concludes the process! Just execute Azure Pipelines, and you will see the magic happen.
For more information, you can contact us at [email protected].
More from our blog
You might also be interested in
