CircleCI orbs are open-source and shareable packages of configuration elements, including jobs, commands, and executors. Using orbs makes customising CircleCI configuration simple.
The key steps to set up the CircleCI Orb integration are –
- Set up the project repository.
- Customize the Provar build.xml file.
- Configure the CircleCI config file.
- How to build a CircleCI Pipeline.
This support article assumes you’ve already integrated your Provar project with a Git repository as documented here. The details provided in this support article are in terms of which artifacts to commit to your Git repository.
The steps in this support article can be completed using a free CircleCI account Sign up is available here, or you can use an existing account.
Note: Currently, the CircleCI can only be used with GitHub or Bitbucket repositories.
The code repositories supported by CircleCI can be found on their integrations page.
Steps to set up the CircleCI Orb integration
The detailed steps to set up the CircleCI integration are given below.
In the example, we have used the following tools to create a Continuous Integration (CI) pipeline as in the screenshot given below. You can adapt this for your own environment.
Step 1: Set up the project repository.
For this step, we will configure the Version Control System (VCS) repository. This is a public repository that contains an example of a CircleCI configuration file and a Provar project. The minimum requirements are as follows (looking at the repository home).
-ProvarProject (containing .testproject, tests folder, and build files) -.settings -ANT -build_circleci.xml // can be named anything, just use this one as an example. -META-INFS -MANIFEST.MF -plans // not required, but our main example uses a build file with a test plan. -Any test plan folders or test plans -tests -Any tests or test folders -templates -Any Test data -src -pageobjects -customapis -.secrets // Make sure your project is encrypted prior to checking in this file. -.testproject -.classpath -build.properties -.project -.circleci -config.yml
Step 2: Customize the Provar build.xml file.
Example build_circleci.xml file (using test plans):
Example build_circleci.xml file (without test plans):
Step 3: Configure the CircleCI config file.
Example of config.yml file:
We suggest that you begin with our build.xml and config.yml files as templates. However, you don’t need to match our example build.xml and config.yml files exactly.
There are some common environment variables and parameters referenced to make things easy. You can modify the following parameters according to your project:
-test_plan: The name of the test plan in the plans/ folder of your Provar project. If you are not using a test plan, then you can leave this parameter blank. For informatIon on test plans, please refer to this page.
-email_target: The email address of the person to send test results/reports.
-build_file: The name of the build file in the ANT/ folder of your Provar Project you want to execute.
-environment: The name of the test environment you want to override your connections/variables within Provar.
-project_name: The name of the folder in your repository home that contains the Provar project.
-test_folder: The name of the test folder in your project that you want to execute (This can be specific or general as you prefer). If you are using a test plan, then you can leave this parameter blank.
Make sure your project is checked into the VCS of your choice before you continue.
Step 4: How to build a CircleCI Pipeline.
Please refer to this Orb link to build a CircleCI pipeline. For this article, you can refer to the run_provar_command example as a base. The steps on how to build your CircleCI pipeline to run Provar tests are given below.
Step 4a: Set up the CircleCI Project.
Navigate to the CircleCI project page to set up your project. If you haven’t already, CircleCI will prompt you to either log in or create a new account. Then, select your organization through GitHub or BitBucket to continue. Find the repository that contains your test project (as configured in the above section) and select Set Up Project.
Step 4b: Build the CircleCI Pipeline.
Select Use Existing Config and click Start Building to initialize the pipeline. If you don’t have a config.yml file contained in the “.circleci” folder of the root of your repository, you’ll need to add one before you can begin building your pipeline.
Note: This will kick-off an initial build immediately if you do have a config.yml file already in your repository. Make sure to add the example config.yml listed above if you do not already have it checked in. The build may fail on the first run, but we still have a few more steps left to configure in the pipeline.
Step 4c: Add the Environment Variables.
Navigate to Project Settings > Environment Variables. You can add project or job-specific environment variables here. In this case, we have created an environment variable for the Provar Secrets password that we used to encrypt the project. We can access environment variables created here, in the build file of our Provar project. In the example repository given above, the build_circleci.xml refers to an environment variable as given below.
We have created a CircleCI Project Environment Variable named ProvarSecretsPassword, we can set the encryption secrets password here. You can also create other optional environment variables for –
- Test plan
- Project name
- Build file
- Test environment
- Email target
Step 4d: Store the Test results.
The last step store_test_results will store the test results in the pipeline details. If your job is successful, you’ll see the results as displayed below.
And, if all of the tests are passing you will see the results as given below.
All these files are displayed as hyperlinks and can be opened directly in CircleCI. For example, you can examine the PDF report by clicking on ProvarProject/ANT/Results/Test_Run_Report.pdf.
Additionally, with the CircleCI insights you can get the historical data on your test suite results as shown below.
CircleCI Common Build Errors
If you come across some build errors, please see CircleCI Common Build Errors for troubleshooting.
- General information
- Licensing Provar
- Provar trial guide and extensions
- Using Provar
- API testing
- Behavior-driven development
- Creating and importing projects
- Creating test cases
- Custom table mapping
- Debugging tests
- Defining a namespace prefix on a connection
- Defining proxy settings
- Environment management
- Exporting test cases into a PDF
- Exporting test projects
- Managing test steps
- Namespace org testing
- Provar desktop
- Provar Test Builder
- Refresh and Recompile
- Reload Org Cache
- Running tests
- Searching Provar with find usages
- Secrets management and encryption
- Setup and teardown test cases
- Tags and Service Level Agreements (SLAs)
- Test cycles
- Test plans
- Testing browser options
- Tooltip testing
- Using the Test Palette
- Test Palette introduction
- Control test steps
- List compare
- Read test step
- String test steps
- UI Test Steps
- Using custom APIs
- Callable tests
- Data-driven testing
- Page objects
- Block locator strategies
- Introduction to XPaths
- Creating an XPath
- Label locator strategies
- Maintaining page objects
- Mapping non-Salesforce fields
- Page object operations
- Refresh and reselect field locators in Test Builder
- Using Java method annotations for custom objects
- Applications testing
- Database testing
- Document testing
- Email testing
- Mobile testing
- OrchestraCMS Testing
- Salesforce CPQ testing
- ServiceMax testing
- Skuid testing
- Vlocity testing
- Webservices testing
- Introduction to test scheduling
- Apache Ant
- Configuration for sending emails via the Provar Command Line Interface (CLI)
- Continuous integration
- Azure DevOps
- Running a Provar CI task in Azure DevOps
- Configuring the Provar secrets password in Microsoft Azure Pipelines
- Parallel execution in Microsoft Azure Pipelines using multiple build.xml files
- Parallel execution in Microsoft Azure Pipelines using targets
- Parallel execution in Microsoft Azure Pipelines using Test Plans
- Bitbucket Pipelines
- GitHub Actions
- Running a Provar CI task in GitHub Actions
- Remote Trigger in GitHub Actions
- Parameterization using Environment Variables in GitHub Actions
- Parallel Execution in GitHub Actions using multiple build.xml files
- Parallel Execution in GitHub Actions using Targets
- Parallel Execution in GitHub Actions using Test Plan
- Parallel Execution in GitHub Actions using Job Matrix
- GitLab CI
- Travis CI
- Execution Environment Security Configuration
- Parallel Execution
- Running Provar on Linux
- Salesforce DX
- Team foundation server
- Version control
- Zephyr Cloud and Server
- Salesforce testing
- Adding a Salesforce connection
- Assert Page Error Messages on Add/Edit Product
- Dynamic Forms
- Internationalization support
- List and table testing
- Salesforce Release Updates
- Salesforce Lightning Testing
- Salesforce Lightning Web Component (LWC) locator support
- Salesforce console testing
- Visualforce Testing
- Testing best practices
- Configurations and permissions
- Error messages
- Licensing, installation and firewalls
- Test Builder and test cases
- Release notes