CircleCI via Docker

More and more enterprise businesses have adopted a DevOps methodology for delivering continuous improvement to their business users such as CircleCI via Docker. The Salesforce platform is at the forefront of this from a SaaS CRM perspective. Salesforce customers use a variety of different CI platforms including both hosted and cloud based platforms to implement their Continuous Improvement and Continuous Deployment pipelines.

Most of Provar’s documentation refers to Jenkins, widely regarded as a top CI/CD tool due to its flexibility to be customized through a wide array of plugins, and its ability to run locally and on cloud. CircleCI is also a top CI tool and extremely popular for its cloud based solution that is easy to setup and configure.

Provar currently runs under any CI tool that can execute an ANT task, which generally means any CI tool that can execute a command line interface. In this guide we’ll walk you through the process of setting up a Continuous Integration Pipeline in CircleCI via Docker container. This forms the first part of Provar’s Containerization roadmap.

This guide will walk you through setting up a CircleCI Pipeline that can be run as a Continuous Integration process when changes are made to your Version Control System (VCS). We do not provide documentation on managing your application deployments but there is plenty of documentation already available for doing this both with Salesforce Scratch Orgs and Sandboxes.

Similar guides to this have been made available for Jenkins, TFS and Azure DevOps (VSTS), see the Further Reading section for more information.

Prerequisites in CircleCI using Docker

The following steps can be completed using a free CircleCI account. Sign up is available here, or you can use an existing account. This guide also assumes you’ve already integrated your Provar project with a Git repository as documented here. The details are provided below in terms of which artifacts to commit to your Git repository.

Note: The CircleCI can only be used with Github or Bitbucket repositories currently.


In the example, we have used the following systems to create a Continuous Integration pipeline.

In the example, we have used the following systems to create a Continuous Integration pipeline.

You can adapt this for your own environment, hosting your VCS repository within Azure DevOps is a simpler solution to the one listed below. The key steps are setup of your project repository, customization of your Provar build.xml file, and configuration of the ANT task. We’ll cover the rest of these steps below.

VCS Repository Configuration

We’ve made public a working project you can use on GitHub, pre-configured for Azure DevOps with the examples includes in the screenshots in this guide:

You may wish to test configuration with this project, creating your own branch, before you attempt to configure your own project. The key project structure is as follows. You can have more files but this is the minimum required.

Repo root

|_ .circleci                                             // The steps below will walk you through config.yaml

|_ config.yml                             // The YAML pipeline script that is executed by CircleCI

|_ DemoProject                                     // Provar Project you wish to test, using our Git plugin

|_ Results                                 // You can move this, but note the path for test results

|_ .secrets                                 // Do not store your .secrets file in a public repository!

|_ src

|_ pageobjects              // Pageobjects locator source files

|_ tests                                     // Parent folder for running all tests if required

|_         // Test cases you wish to execute

|_ .testproject                            // Provar project configuration file

|_ build.xml                               // ANT build.xml file to be executed in the pipeline

|_ Provar

|_ .licenses                                // Folder for your license files, you can have multiple

|_   // Ensure your license file isn’t public

|_ ProvarHome

|_ ant                                        // From our Provar ANT Download

|_ lib                                         // From our Provar ANT Download

For instructions on how to integrate your Provar project into Git please refer to our Git and Provar Projects guide.

Build.xml Configuration

Provar helps you create a default build.xml which is great for running locally, but you will most likely need to adapt this for you CI server environment. The full set of parameters are documented here. Below is an example from the DemoProject with some values called out for your attention. This sample file shows the build.xml below in plaintext.

Setting Notes
project default=”runtests” The default ANT target when no target is specified
property name=”testproject.home” The Provar project root in your repository
property name=”testproject.results” The Provar Results directory in your repository
target name=”runtests” You can, and should, have multiple targets defined in your build.xml. In our example we had one for CircleCI and one for VSTS, but this can be for different Environments, Browsers and/or test levels (Smoke, Regression, Unit Test) for different stages of your DevOps process
testEnvironment=”UAT” Only specify if you’ve defined Environments in your Provar project. These are included in the .testproject file
salesforceMetadataCache=”Reload” Depending on your type of pipeline you can either Reuse metadata cached in the repository or specify to get the latest metadata every time
licensePath=”/home/circleci/project/Provar/.licenses” License path needs to be fully qualified currently. We’re soon releasing a relative path solution to ensure you can run on multiple agents in a pool.
fileset dir=”../DemoProject/tests” You can parameterize what tests to run, or select a suite of tests or folders to include. This will normally vary based on the step in your DevOps process you are testing. Regression tests tend to be a full test pack, but during a developer making a Pull Request you’d run a smaller set of tests to verify critical areas in less time.

CircleCI Pipeline

Follow the steps below to set up your CircleCI Pipeline. You should review the features of CircleCI before you decide how to build your pipelines and what parameters to configure.

Step 1: Create a new project.

Complete the details as per your specific requirements. Projects are synonymous with VCS repositories you have already setup either in GitHub or Atlassian Bitbucket. Pick the Git repo you integrated with Provar already.

CircleCI Pipeline new project window

Step 2: Select VCS Repository Type.

To go any further you need to create a config.yml file and add it to your repository as indicated in the prerequisite steps.

image on how to Select VCS Repository Type

# Provar CircleCI 2.0 configuration file


# Check for more details


version: 2




# Provar 1.8 and 1.9 run using Java 8, amend accordingly if

# using a later version of Provar. The -browsers variant sets up

# all prerequisites for Selenium testing, nice huh?

– image: circleci/openjdk:8-jdk-browsers

# Specify service dependencies here if necessary

# CircleCI maintains a library of pre-built images

# documented at

# No further dependencies required for Provar

# working_directory: ~/repo


# Customize the JVM maximum heap limit

# Set up environment variables here that don’t need to be secure

# We’ll add some additional variables in CircleCI itself for UN & PW

PROVAR_HOME: /home/circleci/project/ProvarHome

testproject.home: /home/circleci/project/DemoProject

testproject.results: /home/circleci/project/DemoProject/Results


# Retrieve the contents of the repo

– checkout

# Download and cache dependencies

– restore_cache:


– v1-dependencies-{{ checksum “DemoProject/build.xml” }}

# fallback to using the latest cache if no exact match is found

– v1-dependencies-

# run: Provar Test Scripts under ANT

# See

# for more info on Provar ANT parameters

– run: ant -lib $PROVAR_HOME/ant -f DemoProject/build.xml

# List test results folder for display

– store_test_results:

path: /home/circleci/project/CIProject/Results

screenshot of configuring VCS Repository Type in CircleCI

Once you’ve checked in your yaml file click the Start Building button.

Step 3: Add missing environment variables.

Unless you included your Provar .secrets file into your Git repository your build might well fail first time. Never fear! We can set the password for our test project Salesforce user securely using CircleCI’s environment variables.

Click the Variables sub-tab and add an entry for the PROVAR_HOME based on your Git repository structure, in our case this is CIProject. This is a useful place for other environment variables you may want to edit frequently, such as which ANT target to execute, the environment or test case path.

how to add missing environment variables in CircleCI

If you’re using a public repository and you have authenticated connections we recommend you also store your connection usernames and passwords as either pipeline or as Release Variables. Provar looks for 2 special environment variables when it cannot find a .secrets file for a connection:

Note that this is case sensitive, ensure you replicate the correct format above.

Note that as of Provar v1.9.5 you can specify your own encryption key for the Provar .secrets file and simply pass the encryption key as an environment variable to the ANT task in your config.yml file. This is also useful when you have multiple SF connections. See Provar Secrets Encryption  for more information.

Step 4: Re-run pipeline.

If all is well your pipeline is now ready to run. Whenever you check-in new tests or edit tests as part of your Github (or Bitbucket) repository the pipeline will trigger a job to execute your tests. You probably don’t want to run your whole test pack every time a single test changes, so consider how you configure your CircleCI Docker project to the only test what’s changed.

Step 5: If all goes well.

You should get a successful job result. It’s highly likely you won’t on the first attempt, so click into the error if you get one and review the issue. Most common causes are incorrect license file location or type, inconsistent settings in build.xml target or missing files from the git repository.

Even your failures tell a story, and drilling down into CircleCI’s test analysis can be interesting to see trends over time.

sample of CircleCI’s test analysis

sample of Job status in CircleCI’s test analysis

Troubleshooting CircleCI Pipeline

After configuring from the steps above your pipeline process should be working 100%. If you’ve deviated from the steps above check carefully the changes you have made are consistent and in synch. (e.g. such as location of build.xml, Provar Home configuration and Provar License files and path).

The following is a summary of common symptoms and how to correct them.

Symptom Cause Solution
No Provar license found Ant job is started but Provar exits if no valid license key is found Move the .license folder to the correct position or rectify the licensePath in your build.xml
License incorrect format The licensePath can be either a folder or specific file. The content of that file must be in the correct format. Activate licenses locally using Provar Desktop and upload the .license files to your repo without editing them
Chrome/Firefox/IE not installed The VSTS agent does not include browser apps Check the Agent Pool selected is Hosted VS2017 or a Custom Build Agent which includes the required browsers
Results not found The file location specific in the ANT job and/or build.xml do not match Check values of both and ensure they are in synch before running again.
Review Provar on G2
Documentation library

Other available resources

Looking for something different?

We use cookies to better understand how our website is used so we can tailor content for you. For more information about the different cookies we use please take a look at our Privacy Policy.

Scroll to Top