Circle CI
More and more enterprise businesses have adopted a DevOps methodology for delivering continuous improvement to their business users. The Salesforce platform is at the forefront of this from a SaaS CRM perspective. Salesforce customers use a variety of different CI platforms including both hosted and cloud based platforms to implement their Continuous Improvement and Continuous Deployment pipelines.
Most of Provar’s documentation refers to Jenkins, widely regarded as a top CI/CD tool due to its flexibility to be customized through a wide array of plugins, and its ability to run locally and on cloud. CircleCI is also a top CI tool and extremely popular for its cloud based solution that is easy to setup and configure.
Provar currently runs under any CI tool that can execute an ANT task, which generally means any CI tool that can execute a command line interface. In this guide we’ll walk you through the process of setting up a Continuous Integration Pipeline in CircleCI using a Docker container. This forms the first part of Provar’s Containerization roadmap.
This guide will walk you through setting up a CircleCI Pipeline that can be run as a Continuous Integration process when changes are made to your Version Control System (VCS). We do not provide documentation on managing your application deployments but there is plenty of documentation already available for doing this both with Salesforce Scratch Orgs and Sandboxes.
Similar guides to this have been made available for Jenkins, TFS and Azure DevOps (VSTS), see the Further Reading section for more information.
Prerequisites
The following steps can be completed using a free CircleCI account. Sign up is available here, or you can use an existing account. This guide also assumes you’ve already integrated your Provar project with a Git repository as documented here. Details are provided below in terms of which artifacts to commit to your Git repository. Note that CircleCI can only be used with Github or Bitbucket repositories currently.
Overview
In the example we have used the following systems to create a Continuous Integration pipeline:
You can adapt this for your own environment, hosting your VCS repository within Azure DevOps is a simpler solution to the one listed below. The key steps are setup of your project repository, customization of your Provar build.xml file, and configuration of the ANT task. We’ll cover the rest of these steps below.
VCS repository configuration
We’ve made public a working project you can use on GitHub, pre-configured for Azure DevOps with the examples includes in the screenshots in this guide: https://github.com/rclark-provar/cidemo
You may wish to test configuration with this project, creating your own branch, before you attempt to configure your own project. The key project structure is as follows. You can have more files but this is the minimum required:
Repo root
|_ .circleci // The steps below will walk you through config.yaml
|_ config.yml // The YAML pipeline script that is executed by CircleCI
|_ DemoProject // Provar Project you wish to test, using our Git plugin
|_ Results // You can move this, but note the path for test results
|_ .secrets // Do not store your .secrets file in a public repository!
|_ src
|_ pageobjects // Pageobjects locator source files
|_ tests // Parent folder for running all tests if required
|_ // Test cases you wish to execute
|_ .testproject // Provar project configuration file
|_ build.xml // ANT build.xml file to be executed in the pipeline
|_ Provar
|_ .licenses // Folder for your license files, you can have multiple
|_ urlicense.properties // Ensure your license file isn’t public
|_ ProvarHome
|_ ant // From our Provar ANT Download
|_ lib // From our Provar ANT Download
For instructions on how to integrate your Provar project into Git please refer to our Git and Provar Projects guide.
Build.xml configuration
Provar helps you create a default build.xml which is great for running locally, but you will most likely need to adapt this for you CI server environment. The full set of parameters are documented here. Below is an example from the DemoProject with some values called out for your attention. This sample file shows the build.xml below in plaintext.
Setting | Notes |
---|---|
project default=”runtests” | The default ANT target when no target is specified |
property name=”testproject.home” | The Provar project root in your repository |
property name=”testproject.results” | The Provar Results directory in your repository |
target name=”runtests” | You can, and should, have multiple targets defined in your build.xml. In our example we had one for CircleCI and one for VSTS, but this can be for different Environments, Browsers and/or test levels (Smoke, Regression, Unit Test) for different stages of your DevOps process |
testEnvironment=”UAT” | Only specify if you’ve defined Environments in your Provar project. These are included in the .testproject file |
webBrowser=”Chrome” | |
salesforceMetadataCache=”Reload” | Depending on your type of pipeline you can either Reuse metadata cached in the repository or specify to get the latest metadata every time |
licensePath=”/home/circleci/project/Provar/.licenses” | License path needs to be fully qualified currently. We’re soon releasing a relative path solution to ensure you can run on multiple agents in a pool. |
fileset dir=”../DemoProject/tests” | You can parameterize what tests to run, or select a suite of tests or folders to include. This will normally vary based on the step in your DevOps process you are testing. Regression tests tend to be a full test pack, but during a developer making a Pull Request you’d run a smaller set of tests to verify critical areas in less time. |
CircleCI pipeline
Follow the steps below to setup your CircleCI Pipeline. You should review the features of CircleCI before you decide how to build your pipelines and what parameters to configure.
Step 1: Create a new project
Complete the details as per your specific requirements. Projects are synonymous with VCS repositories you have already setup either in GitHub or Atlassian Bitbucket. Pick the Git repo you integrated with Provar already.
Step 2: Select VCS Repository Type
To go any further you need to create a config.yml file and add it to your repository as indicated in the prerequisite steps.
# Provar CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-java/ for more details
#
version: 2
jobs:
build:
docker:
# Provar 1.8 and 1.9 run using Java 8, amend accordingly if
# using a later version of Provar. The -browsers variant sets up
# all prerequisites for Selenium testing, nice huh?
– image: circleci/openjdk:8-jdk-browsers
# Specify service dependencies here if necessary
# CircleCI maintains a library of pre-built images
# documented at https://circleci.com/docs/2.0/circleci-images/
# No further dependencies required for Provar
# working_directory: ~/repo
environment:
# Customize the JVM maximum heap limit
# Set up environment variables here that don’t need to be secure
# We’ll add some additional variables in CircleCI itself for UN & PW
PROVAR_HOME: /home/circleci/project/ProvarHome
testproject.home: /home/circleci/project/DemoProject
testproject.results: /home/circleci/project/DemoProject/Results
Steps:
# Retrieve the contents of the repo
– checkout
# Download and cache dependencies
– restore_cache:
keys:
– v1-dependencies-{{ checksum “DemoProject/build.xml” }}
# fallback to using the latest cache if no exact match is found
– v1-dependencies-
# run: Provar Test Scripts under ANT
# See https://www.provartesting.com/support/devops/apache-ant-task-parameters/
# for more info on Provar ANT parameters
– run: ant -lib $PROVAR_HOME/ant -f DemoProject/build.xml
# List test results folder for display
– store_test_results:
path: /home/circleci/project/CIProject/Results
Once you’ve checked in your yaml file click the Start Building button.
Step 3: Add missing environment variables
Unless you included your Provar .secrets file into your Git repository your build might well fail first time. Never fear! We can set the password for our test project Salesforce user securely using CircleCI’s environment variables.
Click the Variables sub-tab and add an entry for the PROVAR_HOME based on your Git repository structure, in our case this is CIProject. This is a useful place for other environment variables you may want to edit frequently, such as which ANT target to execute, the environment or test case path.
If you’re using a public repository and you have authenticated connections we recommend you also store your connection usernames and passwords as either pipeline or as Release Variables. Provar looks for 2 special environment variables when it cannot find a .secrets file for a connection:
Note that this is case sensitive, ensure you replicate the correct format above.
Note that as of Provar v1.9.5 you can specify your own encryption key for the Provar .secrets file and simply pass the encryption key as an environment variable to the ANT task in your config.yml file. This is also useful when you have multiple SF connections. See Provar Secrets Encryption for more information.
Step 4: Re-run pipeline
If all is well your pipeline is now ready to run. Whenever you check-in new tests or edit tests as part of your Github (or Bitbucket) repository the pipeline will trigger a job to execute your tests. You probably don’t want to run your whole test pack every time a single test changes, so consider how you configure your CircleCI project to the only test what’s changed.
Step 5: If all goes well…
You should get a successful job result. It’s highly likely you won’t on the first attempt, so click into the error if you get one and review the issue. Most common causes are incorrect license file location or type, inconsistent settings in build.xml target or missing files from the git repository.
Even your failures tell a story, and drilling down into CircleCI’s test analysis can be interesting to see trends over time:
Troubleshooting
After configuring from the steps above your pipeline process should be working 100%. If you’ve deviated from the steps above check carefully the changes you have made are consistent and in synch. (e.g. such as location of build.xml, Provar Home configuration and Provar License files and path).
The following is a summary of common symptoms and how to correct them:
Symptom | Cause | Solution |
---|---|---|
No Provar license found | Ant job is started but Provar exits if no valid license key is found | Move the .license folder to the correct position or rectify the licensePath in your build.xml |
License incorrect format | The licensePath can be either a folder or specific file. The content of that file must be in the correct format. | Activate licenses locally using Provar Desktop and upload the .license files to your repo without editing them |
Chrome/Firefox/IE not installed | The VSTS agent does not include browser apps | Check the Agent Pool selected is Hosted VS2017 or a Custom Build Agent which includes the required browsers |
Results not found | The file location specific in the ANT job and/or build.xml do not match | Check values of both and ensure they are in synch before running again. |
- General information
- Licensing Provar
- Provar trial guide and extensions
- Using Provar
- API testing
- Behavior-driven development
- Creating and importing projects
- Creating test cases
- Custom table mapping
- Functions
- Debugging tests
- Defining a namespace prefix on a connection
- Defining proxy settings
- Environment management
- Exporting test cases into a PDF
- Exporting test projects
- Managing test steps
- Namespace org testing
- Provar desktop
- Provar Test Builder
- Refresh and Recompile
- Reload Org Cache
- Reporting
- Running tests
- Searching Provar with find usages
- Secrets management and encryption
- Setup and teardown test cases
- Tags and Service Level Agreements (SLAs)
- Test cycles
- Test plans
- Testing browser options
- Tooltip testing
- Using the Test Palette
- Using custom APIs
- Callable tests
- Data-driven testing
- Page objects
- Block locator strategies
- Introduction to XPaths
- Creating an XPath
- JavaScript locator support
- Label locator strategies
- Maintaining page objects
- Mapping non-Salesforce fields
- Page object operations
- ProvarX™
- Refresh and reselect field locators in Test Builder
- Using Java method annotations for custom objects
- Applications testing
- DevOps
- Introduction to test scheduling
- Apache Ant
- Configuration for sending emails via the Provar Command Line Interface (CLI)
- Continuous integration
- AutoRABIT
- Azure DevOps
- Running a Provar CI task in Azure DevOps
- Configuring the Provar secrets password in Microsoft Azure Pipelines
- Parallel execution in Microsoft Azure Pipelines using multiple build.xml files
- Parallel execution in Microsoft Azure Pipelines using targets
- Parallel execution in Microsoft Azure Pipelines using Test Plans
- Bitbucket Pipelines
- Circle CI
- Copado
- Docker
- Flosum
- Gearset
- GitHub Actions
- Running a Provar CI task in GitHub Actions
- Remote Trigger in GitHub Actions
- Parameterization using Environment Variables in GitHub Actions
- Parallel Execution in GitHub Actions using multiple build.xml files
- Parallel Execution in GitHub Actions using Targets
- Parallel Execution in GitHub Actions using Test Plan
- Parallel Execution in GitHub Actions using Job Matrix
- GitLab CI
- Jenkins
- Travis CI
- Parallel Execution
- Running Provar on Linux
- Reporting
- Salesforce DX
- Git
- Team foundation server
- Version control
- Zephyr Cloud and Server
- Salesforce testing
- Adding a Salesforce connection
- Assert Page Error Messages on Add/Edit Product
- Dynamic Forms
- Internationalization support
- List and table testing
- Salesforce Release Updates
- Salesforce Lightning Testing
- Salesforce Lightning Web Component (LWC) locator support
- Salesforce console testing
- Visualforce Testing
- Testing best practices
- Troubleshooting
- Release notes