Running a Provar CI task in Azure DevOps

This guide will walk you through setting up a Microsoft Azure Pipeline that can be run on either a scheduled basis (for daily regression) or as a continuous Integration process when changes are made to your Version Control System (VCS).

Similar guides to this have been made available for Jenkins, TFS and CircleCI, see the Further Reading section for more information.

Prerequisites

The following steps can be completed using a free Azure DevOps account. Signup is available here, or you can use an existing Microsoft account. This guide also assumes you’ve already integrated your Provar project with a git repository. Details are provided below in terms of which artifacts to commit to your VCS repository.

Overview

You can adapt this for your own environment by hosting your VCS repository within Azure DevOps, which may provide a more streamlined solution in comparison to the one listed below. The key steps include: setting up your project repository, customizing your Provar build.xml file and configuring the ANT task. We’ll cover the rest of these steps below in more detail.

VCS repository configuration

We’ve made public a working project that you can use on GitHub. It is preconfigured for Azure DevOps.: https://github.com/rclark-provar/cidemo

You may wish to test the configuration with this project and create your own branch before you attempt to configure your own project. The key project structure is as follows. You can have more files though this is the minimum required:

Repo root
|_ DemoProject
|_ Results |_ .secrets |_ src
|_ pageobjects |_ tests
|_ <files & folders> |_ .testproject
|_ build.xml
|_ Provar
|_ .licenses
// Provar Project you wish to test, using our Git plugin// You can move this, but note the path for test results // Do not store your .secrets file in a public repository!
// Pageobjects locator source files
// Parent folder for running all tests if required
// Test cases you wish to execute
// Provar project configuration file
// ANT build.xml file to be executed in the pipeline
// Folder for your license files, you can have multiple// Ensure your license file isn’t public
// From our Provar ANT Download // From our Provar ANT Download
|_ ProvarHome |_ ant
|_ lib
|_ urlicense.properties

For instructions on how to integrate your Provar project into Git please refer to our support articles about Git.

Build.xml configuration

Provar helps you create a default build.xml which is great for running locally, but you will most likely need to adapt this for your CI server environment. Below is an example from the demo project with some values called out for your attention.

<project default="runtests">

<property environment="env"/>
<property name="provar.home" value="${env.PROVAR_HOME}"/>
<property name="testproject.home" value="DemoProject"/>
<property name="testproject.results" value="DemoProject/Results"/>
<property name="license.path" value="../Provar/.license"/>

     <taskdef name="Provar-Compile"
            classname="com.provar.testrunner.ant.CompileTask"
            classpath="${provar.home}/ant/ant-provar.jar"/>

     <taskdef name="Run-Test-Case"
            classname="com.provar.testrunner.ant.RunnerTask"
            classpath="${provar.home}/ant/ant-
provar.jar;${provar.home}/ant/ant-provar-bundled.jar;${provar.home}/ant/ant-
provar-sf.jar"/>

     <taskdef name="Test-Cycle-Report"
            classname="com.provar.testrunner.ant.TestCycleReportTask"
            classpath="${provar.home}/ant/ant-
provar.jar;${provar.home}/ant/ant-provar-bundled.jar;${provar.home}/ant/ant-
provar-project.jar;${provar.home}/ant/antprovar-sf.jar"/>

     <target name="runtestsVSTS">

           <Provar-Compile
            provarHome="${provar.home}"
            projectPath="${testproject.home}"/>

           <Run-Test-Case 
                 provarHome="${provar.home}"
                 projectPath="${testproject.home}"
                 resultsPath="${testproject.results}"
                 resultsPathDisposition="Replace"
                 testEnvironment="UAT"              
                 webBrowser="Chrome"
                 webBrowserConfiguration="Full Screen"
                 excludeCallableTestCases="true"
                 salesforceMetadataCache="Reload"
                 projectCachePath=".provarCaches"  
                 testOutputlevel="WARNING"
                 licensePath="${license.path}"
                 pluginOutputlevel="WARNING" 
                 stopTestRunOnError="false"
                 invokeTestRunMonitor="true">
           <fileset dir="../DemoProject/tests">
                 <include name="Test Case 1.testcase"/>
            </fileset>
     </Run-Test-Case>
</target>

Azure DevOps Pipelines

Follow the steps below to set up your Azure DevOps Pipelines. You may want to create separate pipelines for different environments. You can also parameterize the pipeline to control the different environments and VCS branches you want to run tests from.

Step 1: Create a new project. Complete the details as per your specific requirements.

Step 2: Import the project in Azure Git (optional).

If you don’t have a repository already, you can let Azure DevOps create one for you. You’ll need to go back and configure this in Provar and commit your test cases, build.xml and Provar ANT files as specified in the previous section.

Also, you can import your repository from any other GIT or TFS repository.

Step 3: Create a new pipeline.

If you don’t want to push your project into Azure git (mentioned in Step 2), you can directly select the Pipelines option and click on Create Pipeline.

Step 4: Select the type of pipeline.

Azure DevOps provides us with two options here. First, create a pipeline using a YAML file. Second, use the classic Editor (without YAML). As there are many options available, you can choose any of them according to your requirements.

Step 5: Using the Classic Editor.

Click on the link Use the classic editor and you will be redirected to select your repository source.

Step 6: Select repository source.

We’ve chosen to use a GitHub repo. You’re free to make whatever selection fits your environment and tools available. Note that the selection here may affect the screens afterwards, but the key items for any Git host are the repository to be used and the default branch.

Step 7: Select the repository and default branch.

When you click Save and Queue this will control the branch used to build and run your tests. You will probably be following a VCS and DevOps strategy where Master is not the branch you build from.

Step 8: Choose the task template.

There are numerous options for how to run Provar test executions. The simplest option is to use the ANT task as this mimics how you can test your regression or CI targets locally in Provar Desktop.

Step 9: Configure ANT task.

It’s important to use a build agent that includes browser support. VS2017-win2016 is what we have used or you can build your own custom agent.

Step 10: Configure pipeline source.

Check that the Repository and Branch are correct. You may have to reselect these. You may have multiple tasks in your pipeline that you access different branches for such as Dev, UAT, Staging, etc.

Step 11: Download Provar binaries.

In case provar binaries are not checked-in in the GIT repository, you need to download it before executing the ANT task.

SInce we have used a windows agent, we can include a Powershell Script task and download Provar binaries from the Provar_ANT.zip file and extract them in the ProvarHome directory.

Commands are mentioned below.

$webclient = New-Object System.Net.WebClient
$url = "https://download.provartesting.com/latest/Provar_ANT_latest.zip"
$file = "Provar_ANT_latest.zip"

$webclient.DownloadFile($url,$file)
Expand-Archive -LiteralPath Provar_ANT_latest.zip -DestinationPath ProvarHome

Step 12: Configure the ANT task.

These values will be specific to your Provar project. Integrating yourJunit results will allow you to see the test result history graphically.

To use JDK 8, go to advanced settings and select JDK 8 in JDK version dropdown.

Step 13: Configure environment variables.

Click the Variables sub-tab and add an entry for PROVAR_HOME based on your VCS repository structure. This is a useful place for other environment variables you may want to edit frequently such as which ANT target to execute, the environment or test path.

If you’re using a public repository and you have authenticated connections, we recommend that you also store your connection usernames and passwords as either pipeline or as release variables. Provar looks for two special (case sensitive) environment variables when it cannot find a .secrets file for a connection.

PROVAR_sf_<connection-name>

PROVAR_sf_<connection-name>_password

In the example below, a Salesforce connection called DemoAdmin has been secured outside of using Github.

Use the lock to secure the credential so it’s not in plain text. The environment variable must be named. You can see how we do this for Jenkins in this password masking demo video.

Step 14: Save and queue to test your pipeline.

Before we trigger the pipeline automatically, first troubleshoot any issues with the setup manually.

Step 15: If all goes well…

You should get a successful job result. It’s highly likely you won’t on the first attempt, so click into the error if you get one and review the issue. The most common causes are incorrect license file location or type, inconsistent settings in the build.xml target or missing files from the VCS repository.

You can also view your history of test results under the test tab and drill-down into the test results to a high level.

Step 16: Provar test results in Azure DevOps.

To get Provar test results (PDF or HTML files), you need to add a task – Publish Artifacts. This will require you to provide the path of the results directory and the artifact name. We have used some pre-defined parameters available in Azure Devops. See more information in this link for Use Predefined Variables.

Step 17: Caching in Azure DevOps.

To reduce the execution time in between the jobs, we can include a caching task in the job. Two parameters are required in caching tasks – Key and Path. Key can be any value and Path would be the relative path of the .provarCaches folder that is defined in build.xml.

Values in build.xml should be:

salesforceMetadataCache=”Reuse”
projectCachePath=”.provarCaches”

Configure CI job

To enable continuous integration, edit your pipeline job and select the Triggers tab. You can check the Continuous Integration flag to make the job run whenever your VCS branch is updated. We also recommend checking the Batch changes option if you want to optimize your pipeline so that it’s not running for each separate commit.

You can also use the Pull Request option if you want to invoke your CI process when a Pull Request is made. This is useful for validating when a developer has changes pending for review and you want to run a set of Provar unit or smoke tests.

Schedule the regression job

In addition to running a CI process, it’s usually best practice to configure a nightly build to execute a full regression pack. You should do this on a separate pipeline to the CI job as you’ll most likely be executing a test pack that takes hours instead of minutes. You will also almost certainly wish to execute this on a different branch of your repository to the CI job.

Executing Provar Tests using YAML file on Linux

If you are trying to configure a job using a YAML file, the steps will be little different but the concepts remain the same.

Below is a sample YAML file that includes the following commands:

Step 1: Trigger.

Continuous integration (CI) triggers a pipeline to run whenever you push an update to the specified branches or push specified tags.

Step 2: Pool.

To build your code or deploy your software using Azure Pipelines, you need at least one agent. Azure Pipelines provides a pre-defined agent pool named Azure Pipelines with Microsoft-hosted agents.

Step 3: Cache.

Pipeline caching can help reduce build time by allowing the outputs or downloaded dependencies from one run to be reused in later runs, thereby reducing or avoiding the cost to recreate or redownload the same files again.

Caching .provarCaches: Execution time required to download metadata on every run can be reduced by caching it.
Caching ProvarHome: Caching of Provar Home directory is done in this step.

Step 4: Bash.

Bash script is used to download Provar Binaries if not already available through cache.

Step 5: ANT Task Execution.

Executing the Ant task using xvfb-run command using the bash task. This task will execute provar test cases and generate test results.

Step 6: Publish your test results.

We have to explicitly use the publish test results task to publish JUNIT test results in GUI.

Step 7: Publish artifacts.

To get Provar Test Results (PDF or HTML files), you need to add a task Publish Artifacts.

Note: To execute on the Linux environment, we need to change the Browser to Chrome_Headless in the build.xml.

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

steps:
- task: [email protected]
  inputs:
    key: 'provarcache'
    path: '$(System.DefaultWorkingDirectory)/.provarCaches'

- task: [email protected]
  inputs:
    key: 'provarhome'
    path: '$(System.DefaultWorkingDirectory)/Provar_Home'

- task: [email protected]
  inputs:
    targetType: 'inline'
    script: |
      if [ ! -d Provar_Home ]; then
        curl -O https://download.provartesting.com/latest/Provar_ANT_latest.zip
        unzip Provar_ANT_latest.zip -d ProvarHome;
      else 
        echo "Provar_Home is already available.";
      fi

- task: [email protected]
  inputs:
    targetType: 'inline'
    script: 'xvfb-run ant -Dprovar.home=$(PROVARHOME) -f DemoProjects/build.xml'

- task: [email protected]
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: 'DemoProjects/Results/*.xml'

- task: [email protected]
  inputs:
    PathtoPublish: 'DemoProjects/ANT/Results'
    ArtifactName: 'ResultsDirectory_$(Build.BuildNumber)'

Troubleshooting

After configuring the steps above, your pipeline process should be working 100%. If you’ve deviated from the steps above please check carefully that the changes you have made are consistent and in sync (for example the location of the build.xml, Provar Home configuration and the Provar license files and path).

The following is a summary of common symptoms and how to correct them: