Running a Provar CI Task in Azure DevOps Pipelines
This guide will walk you through setting up a Microsoft Azure DevOps Pipeline that can be run on either a scheduled basis (for daily regression) or as a continuous Integration process when changes are made to your Version Control System (VCS).
Similar guides to this have been made available for Jenkins, TFS and CircleCI, see the Further Reading section for more information.
Prerequisites in Azure DevOps Pipelines
The following steps can be completed using a free Azure DevOps Pipelines account. Signup is available here, or you can use an existing Microsoft account. This guide also assumes you’ve already integrated your Provar project with a git repository. Details are provided below in terms of which artifacts to commit to your VCS repository.
Overview
You can adapt this for your own environment by hosting your VCS repository within Azure DevOps Pipelines, which may provide a more streamlined solution in comparison to the one listed below. The key steps include: setting up your project repository, customizing your Provar build.xml file and configuring the ANT task. We’ll cover the rest of these steps below in more detail.
VCS Repository Configuration
We’ve made public a working project that you can use on GitHub. It is preconfigured for Azure DevOps.: https://github.com/rclark-provar/cidemo
You may wish to test the configuration with this project and create your own branch before you attempt to configure your own project. The key project structure is as follows. You can have more files though this is the minimum required:
Repo root |_ DemoProject |_ Results |_ .secrets |_ src |_ pageobjects |_ tests |_ <files & folders> |_ .testproject |_ build.xml |_ Provar |_ .licenses // Provar Project you wish to test, using our Git plugin// You can move this, but note the path for test results // Do not store your .secrets file in a public repository! // Pageobjects locator source files // Parent folder for running all tests if required // Test cases you wish to execute // Provar project configuration file // ANT build.xml file to be executed in the pipeline // Folder for your license files, you can have multiple// Ensure your license file isn’t public // From our Provar ANT Download // From our Provar ANT Download |_ ProvarHome |_ ant |_ lib |_ urlicense.properties
For instructions on how to integrate your Provar project into Git please refer to our support articles about Git.
Build.xml Configuration
Provar helps you create a default build.xml which is great for running locally, but you will most likely need to adapt this for your CI server environment. Below is an example from the demo project with some values called out for your attention.
<project default="runtests"> <property environment="env"/> <property name="provar.home" value="${env.PROVAR_HOME}"/> <property name="testproject.home" value="DemoProject"/> <property name="testproject.results" value="DemoProject/Results"/> <property name="license.path" value="../Provar/.license"/> <taskdef name="Provar-Compile" classname="com.provar.testrunner.ant.CompileTask" classpath="${provar.home}/ant/ant-provar.jar"/> <taskdef name="Run-Test-Case" classname="com.provar.testrunner.ant.RunnerTask" classpath="${provar.home}/ant/ant- provar.jar;${provar.home}/ant/ant-provar-bundled.jar;${provar.home}/ant/ant- provar-sf.jar"/> <taskdef name="Test-Cycle-Report" classname="com.provar.testrunner.ant.TestCycleReportTask" classpath="${provar.home}/ant/ant- provar.jar;${provar.home}/ant/ant-provar-bundled.jar;${provar.home}/ant/ant- provar-project.jar;${provar.home}/ant/antprovar-sf.jar"/> <target name="runtestsVSTS"> <Provar-Compile provarHome="${provar.home}" projectPath="${testproject.home}"/> <Run-Test-Case provarHome="${provar.home}" projectPath="${testproject.home}" resultsPath="${testproject.results}" resultsPathDisposition="Replace" testEnvironment="UAT" webBrowser="Chrome" webBrowserConfiguration="Full Screen" excludeCallableTestCases="true" salesforceMetadataCache="Reload" projectCachePath=".provarCaches" testOutputlevel="WARNING" licensePath="${license.path}" pluginOutputlevel="WARNING" stopTestRunOnError="false" invokeTestRunMonitor="true"> <fileset dir="../DemoProject/tests"> <include name="Test Case 1.testcase"/> </fileset> </Run-Test-Case> </target>
Azure DevOps Pipelines
Follow the steps below to set up your Azure DevOps Pipelines. You may want to create separate pipelines for different environments. You can also parameterize the pipeline to control the different environments and VCS branches you want to run tests from.
Step 1: Create a new project. Complete the details as per your specific requirements.
Step 2: Import the project in Azure Git (optional).
If you don’t have a repository already, you can let Azure DevOps create one for you. You’ll need to go back and configure this in Provar and commit your test cases, build.xml and Provar ANT files as specified in the previous section.
Also, you can import your repository from any other GIT or TFS repository.
Step 3: Create a new pipeline.
If you don’t want to push your project into Azure git (mentioned in Step 2), you can directly select the Pipelines option and click on Create Pipeline.
Step 4: Select the type of pipeline.
Azure DevOps provides us with two options here. First, create a pipeline using a YAML file. Second, use the classic Editor (without YAML). As there are many options available, you can choose any of them according to your requirements.
Step 5: Using the Classic Editor.
Click on the link Use the classic editor and you will be redirected to select your repository source.
Step 6: Select repository source.
We’ve chosen to use a GitHub repo. You’re free to make whatever selection fits your environment and tools available. Note that the selection here may affect the screens afterwards, but the key items for any Git host are the repository to be used and the default branch.
Step 7: Select the repository and default branch.
When you click Save and Queue this will control the branch used to build and run your tests. You will probably be following a VCS and DevOps strategy where Master is not the branch you build from.
Step 8: Choose the task template.
There are numerous options for how to run Provar test executions. The simplest option is to use the ANT task as this mimics how you can test your regression or CI targets locally in Provar Desktop.
Step 9: Configure ANT task.
It’s important to use a build agent that includes browser support. VS2017-win2016 is what we have used or you can build your own custom agent.
Step 10: Configure pipeline source.
Check that the Repository and Branch are correct. You may have to reselect these. You may have multiple tasks in your pipeline that you access different branches for such as Dev, UAT, Staging, etc.
Step 11: Download Provar binaries.
In case provar binaries are not checked-in in the GIT repository, you need to download it before executing the ANT task.
SInce we have used a windows agent, we can include a Powershell Script task and download Provar binaries from the Provar_ANT.zip file and extract them in the ProvarHome directory.
Commands are mentioned below.
$webclient = New-Object System.Net.WebClient $url = "https://download.provartesting.com/latest/Provar_ANT_latest.zip" $file = "Provar_ANT_latest.zip" $webclient.DownloadFile($url,$file) Expand-Archive -LiteralPath Provar_ANT_latest.zip -DestinationPath ProvarHome
Step 12: Configure the ANT task.
These values will be specific to your Provar project. Integrating yourJunit results will allow you to see the test result history graphically.
To use JDK 8, go to advanced settings and select JDK 8 in JDK version dropdown.
Step 13: Configure environment variables.
Click the Variables sub-tab and add an entry for PROVAR_HOME based on your VCS repository structure. This is a useful place for other environment variables you may want to edit frequently such as which ANT target to execute, the environment or test path.
If you’re using a public repository and you have authenticated connections, we recommend that you also store your connection usernames and passwords as either pipeline or as release variables. Provar looks for two special (case sensitive) environment variables when it cannot find a .secrets file for a connection.
PROVAR_sf_<connection-name>
PROVAR_sf_<connection-name>_password
In the example below, a Salesforce connection called DemoAdmin has been secured outside of using Github.
Use the lock to secure the credential so it’s not in plain text. The environment variable must be named. You can see how we do this for Jenkins in this password masking demo video.
Step 14: Save and queue to test your pipeline.
Before we trigger the pipeline automatically, first troubleshoot any issues with the setup manually.
Step 15: If all goes well…
You should get a successful job result. It’s highly likely you won’t on the first attempt, so click into the error if you get one and review the issue. The most common causes are incorrect license file location or type, inconsistent settings in the build.xml target or missing files from the VCS repository.
You can also view your history of test results under the test tab and drill-down into the test results to a high level.
Step 16: Provar test results in Azure DevOps.
To get Provar test results (PDF or HTML files), you need to add a task – Publish Artifacts. This will require you to provide the path of the results directory and the artifact name. We have used some pre-defined parameters available in Azure Devops. See more information in this link for Use Predefined Variables.
Step 17: Caching in Azure DevOps.
To reduce the execution time in between the jobs, we can include a caching task in the job. Two parameters are required in caching tasks – Key and Path. Key can be any value and Path would be the relative path of the .provarCaches folder that is defined in build.xml.
Values in build.xml should be:
salesforceMetadataCache=”Reuse” projectCachePath=”.provarCaches” |
Configure CI Job
To enable continuous integration, edit your pipeline job and select the Triggers tab. You can check the Continuous Integration flag to make the job run whenever your VCS branch is updated. We also recommend checking the Batch changes option if you want to optimize your pipeline so that it’s not running for each separate commit.
You can also use the Pull Request option if you want to invoke your CI process when a Pull Request is made. This is useful for validating when a developer has changes pending for review and you want to run a set of Provar unit or smoke tests.
Schedule the Regression Job
In addition to running a CI process, it’s usually best practice to configure a nightly build to execute a full regression pack. You should do this on a separate pipeline to the CI job as you’ll most likely be executing a test pack that takes hours instead of minutes. You will also almost certainly wish to execute this on a different branch of your repository to the CI job.
Executing Provar Tests using YAML file on Linux
If you are trying to configure a job using a YAML file, the steps will be little different but the concepts remain the same.
Below is a sample YAML file that includes the following commands:
Step 1: Trigger.
Continuous integration (CI) triggers a pipeline to run whenever you push an update to the specified branches or push specified tags.
Step 2: Pool.
To build your code or deploy your software using Azure Pipelines, you need at least one agent. Azure Pipelines provides a pre-defined agent pool named Azure Pipelines with Microsoft-hosted agents.
Step 3: Cache.
Pipeline caching can help reduce build time by allowing the outputs or downloaded dependencies from one run to be reused in later runs, thereby reducing or avoiding the cost to recreate or redownload the same files again.
- Caching .provarCaches: Execution time required to download metadata on every run can be reduced by caching it.
- Caching ProvarHome: Caching of Provar Home directory is done in this step.
Step 4: Bash.
Bash script is used to download Provar Binaries if not already available through cache.
Step 5: ANT Task Execution.
Executing the Ant task using xvfb-run command using the bash task. This task will execute provar test cases and generate test results.
Step 6: Publish your test results.
We have to explicitly use the publish test results task to publish JUNIT test results in GUI.
Step 7: Publish artifacts.
To get Provar Test Results (PDF or HTML files), you need to add a task Publish Artifacts.
Note: To execute on the Linux environment, we need to change the Browser to Chrome_Headless in the build.xml.
trigger: - master pool: vmImage: 'ubuntu-latest' steps: - task: [email protected] inputs: key: 'provarcache' path: '$(System.DefaultWorkingDirectory)/.provarCaches' - task: [email protected] inputs: key: 'provarhome' path: '$(System.DefaultWorkingDirectory)/Provar_Home' - task: [email protected] inputs: targetType: 'inline' script: | if [ ! -d Provar_Home ]; then curl -O https://download.provartesting.com/latest/Provar_ANT_latest.zip unzip Provar_ANT_latest.zip -d ProvarHome; else echo "Provar_Home is already available."; fi - task: [email protected] inputs: targetType: 'inline' script: 'xvfb-run ant -Dprovar.home=$(PROVARHOME) -f DemoProjects/build.xml' - task: [email protected] inputs: testResultsFormat: 'JUnit' testResultsFiles: 'DemoProjects/Results/*.xml' - task: [email protected] inputs: PathtoPublish: 'DemoProjects/ANT/Results' ArtifactName: 'ResultsDirectory_$(Build.BuildNumber)'
Troubleshooting
After configuring the steps above, your pipeline process should be working 100%. If you’ve deviated from the steps above please check carefully that the changes you have made are consistent and in sync (for example the location of the build.xml, Provar Home configuration and the Provar license files and path).
The following is a summary of common symptoms and how to correct them:
Further Reading
The following articles may also be of interest:
Introduction to test scheduling
Introduction to Git integration
- General information
- Licensing Provar
- Provar trial guide and extensions
- Using Provar
- API testing
- Behavior-driven development
- Creating and importing projects
- Creating test cases
- Custom table mapping
- Functions
- Debugging tests
- Defining a namespace prefix on a connection
- Defining proxy settings
- Environment management
- Exporting test cases into a PDF
- Exporting test projects
- Override auto-retry for Test Step
- Managing test steps
- Namespace org testing
- Provar desktop
- Provar Test Builder
- Refresh and Recompile
- Reload Org Cache
- Reporting
- Running tests
- Searching Provar with find usages
- Secrets management and encryption
- Setup and teardown test cases
- Tags and Service Level Agreements (SLAs)
- Test cycles
- Test plans
- Testing browser options
- Tooltip testing
- Using the Test Palette
- Using custom APIs
- Callable tests
- Data-driven testing
- Page objects
- Block locator strategies
- Introduction to XPaths
- Creating an XPath
- JavaScript locator support
- Label locator strategies
- Maintaining page objects
- Mapping non-Salesforce fields
- Page object operations
- ProvarX™
- Refresh and reselect field locators in Test Builder
- Using Java method annotations for custom objects
- Applications testing
- DevOps
- Introduction to test scheduling
- Apache Ant
- Configuration for Sending Emails via the Provar Command Line Interface
- Continuous integration
- AutoRABIT Salesforce DevOps in Provar Test
- Azure DevOps
- Running a Provar CI Task in Azure DevOps Pipelines
- Configuring the Provar secrets password in Microsoft Azure Pipelines
- Parallel Execution in Microsoft Azure Pipelines Using Multiple build.xml Files
- Parallel Execution in Microsoft Azure Pipelines using Targets
- Parallel execution in Microsoft Azure Pipelines using Test Plans
- Bitbucket Pipelines
- CircleCI
- Copado
- Docker
- Flosum
- Gearset DevOps CI/CD
- GitHub Actions
- Integrating GitHub Actions CI to Run Provar CI Task
- Remote Trigger in GitHub Actions
- Parameterization using Environment Variables in GitHub Actions
- Parallel Execution in GitHub Actions using Multiple build.xml Files
- Parallel Execution in GitHub Actions using Targets
- Parallel Execution in GitHub Actions using Test Plan
- Parallel Execution in GitHub Actions using Job Matrix
- GitLab Continuous Integration
- Travis CI
- Jenkins
- Execution Environment Security Configuration
- Provar Jenkins Plugin
- Parallel Execution
- Running Provar on Linux
- Reporting
- Salesforce DX
- Git
- Team foundation server
- Version control
- Salesforce testing
- Adding a Salesforce connection
- Assert Page Error Messages on Add/Edit Product
- Dynamic Forms
- Internationalization support
- List and table testing
- Salesforce Release Updates
- Salesforce Lightning Testing
- Salesforce Lightning Web Component (LWC) locator support
- Salesforce console testing
- Visualforce Testing
- Performance Best Practices
- Testing best practices
- Troubleshooting
- Browsers
- Configurations and permissions
- Connections
- DevOps
- Error messages
- Administrator has blocked access to client
- JavascriptException: Javascript error
- macOS Big Sur upgrade issue
- Resolving failed to create ChromeDriver error
- Resolving Jenkins license missing error
- Resolving metadata timeout errors
- Test execution fails – Firefox not installed
- Update to Opportunity field validation behaviour
- Licensing, installation and firewalls
- Memory
- Settings
- Test Builder and test cases
- Release notes