Salesforce DX integration
Provar and Salesforce DX can be used together to generate a scratch org, push the latest Salesforce customizations and run Provar tests inside the scratch org to get functional testing insights. This allows immediate feedback during a development cycle or when a scratch org is used as a system integration test environment.
As part of our ongoing commitment as the only Salesforce-first test automation tool we support integration of Provar with the Salesforce CLI using ProvarDX™. For information about how you can run Provar tests for Salesforce DX scratch orgs using ANT and ConfigureDX, please see this related article.
ProvarDX™ (Provar Developer eXperience) is a free, open-source extension that makes use of your local Provar installation licenses. ProvarDX™ is designed to integrate Provar seamlessly into your Salesforce developer environments so that tests can be executed supporting local development using scratch orgs. This provides a richer set of tools than ConfigureDX, which is designed purely for customers running Provar tests on scratch orgs under ANT.
ProvarDX™ consists of the following elements
- A Salesforce CLI plugin that allows you to execute Provar test cases from a command line without requiring the installation or use of ANT. ProvarDX™ can be used instead of ANT as part of your CI/CD pipeline in order to run Provar tests.
- An optional Visual Studio Code Extension that provides access to the most useful ProvarDX™ CLI commands using the Visual Studio Code command autocomplete menu.
- A Provar Desktop right-click option (Export as ProvarDX™) to allow you to export one or more tests (or test plans) as a property file for execution using ProvarDX™.
Both ANT (via ConfigureDX) and ProvarDX™ support the use of scratch orgs for test execution. However, ProvarDX™ has a number of advantages including:
- The ability to override Provar Salesforce connections using the DevHub org alias as the connection name instead of requiring you to use the original Salesforce username as the alias. This subtle change allows you to be very clear in your scratch org names and use meaningful aliases (such as Sales User, Service Admin, Metadata Admin etc.) instead of [email protected]
- ProvarDX™ works on both scratch orgs and sandboxes, allowing you to use a single solution regardless of whether you follow a source-driven, hybrid or org-driven Salesforce application lifecycle.
- ProvarDX™ does not require you to generate passwords for your scratch orgs. ProvarDX™ does this for you when it needs one.
- With ProvarDX™, you can override property settings on command line execution using parameters.
- ProvarDX™ is consistent with the Salesforce CLI, requiring a single set of skills to maintain and operate. Updates are also synchronized with the Salesforce CLI.
Everything in this guide applies to teams following source-driven development using scratch orgs and those following org-driven development using sandbox orgs.
Provar is designed to help our customers quickly create robust, reusable tests that are intelligent regarding Salesforce org metadata and reusable across multiple user types, environments and even user interfaces.
Provar is uniquely able to run across multiple environments used in a typical software development lifecycle. In order to align with Salesforce’s source-driven development approach using scratch orgs, and traditional org-driven development, we’ve delivered ProvarDX™ to make this integration seamless for our users.
- Admins and developers using scratch orgs and/or developer sandboxes to make atomic changes which they want to verify before pushing to continuous integration.
- CI tools hooked into a version control system (e.g. Git) triggers publication and runs a set of Provar regression tests to validate the changes. Issues notified to the development and QA team for correction or update of test cases to match changes in behavior.
- CD pipeline invoked as needed to deploy application changes to each environment. Provar regression and smoke test plans executed as appropriate to validate correct deployment package deployment plus check for legacy issues or primary business scenarios are working as expected.
As you can see, with ProvarDX™ we’re unlocking those test cases already developed using Provar so that Salesforce administrators and developers can run a suite of tests to verify their changes much earlier in the development process. We’ve chosen to do this using the exact same tooling that Salesforce provides, Visual Source Code and the Salesforce Command Line Interface (CLI).
Note: This does not eliminate the need for developers to write unit tests for their apex code and Lightning components.
To utilize ProvarDX™, you need to first install the Salesforce CLI (following Salesforce’s installation recommendations) in the environment where ProvarDX™ will be executed.
After installing the Salesforce CLI, the ProvarDX™ package can be installed which extends the Salesforce CLI command set with four simple commands using the Provar namespace:
Runs a specific set of tests using environment overrides just as you would using the ANT build.xml file. Like ANT, we support the same muti-browser and test reporting options. Behavior is controlled by the provardx property file. We recommend having multiple provardx properties to reflect each environment you want to test and the appropriate test plan for that environment.
Optional command to allow you to quickly validate changes you make to your ProvarDX™ property files rather than wait until you try and use runtests to execute them.
Downloads the latest metadata for the named Salesforce connections and stores this locally. This optional command allows you to preprocess this step rather than wait during your runtests command for any changes to download. Very useful for CI/CD pipelines where you want to execute post-deployment steps before starting a full test run.
Optional command to allow you to precompile any Provar page objects rather than wait until your runtests command is executed. Useful for ensuring your Provar PageObjects, Custom APIs and PageControls before starting a test run. Typically used in a CI/CD pipeline after pulling the latest Provar project from your version control system.
Each of these commands has a number of flags and options you can set to control their operation. Each flag has a longform (e.g. –targetusername) and short form (e.g. -u) for convenience. Please see the ProvarDX™ NPM listing to get the up-to-date list of flags and options.
Note: If you’re struggling to remember the correct command syntax, just type
sfdx provar --help to see a list of the options and parameters. You can do this at any level of the command, e.g.
sfdx provar:runtests --help tells you the parameters of the
In addition to the Salesforce CLI extension, we have also made available a Visual Studio Code Extension for those who prefer to use the Visual Studio Code command palette window. These commands can be accessed just like any other Visual Studio Code command using either Command + P (Mac OS) or Ctrl + P (Windows).
We currently provide three commands via the Visual Studio Code extension
ProvarDX: Create Properties File
Creates a boilerplate provardx-properties.json file so you just have to edit the values you need.
ProvarDX: Run Testcases
sfdx:provar:runtests using a specified provardx property file.
ProvarDX: Validate Properties File
sfdx:provar:validate for the named provardx property file.
We plan to add more commands in the future, so please bookmark this page and watch out for upcoming Provar release notes for announcements on ProvarDX™ enhancements.
Step 1: Using Provar Desktop first open your Provar project.
Step 1: Within the Navigator, right-click on the test plan, test case folder or the individual test cases you want to run under ProvarDX™.
Step 3: Select the Export as ProvarDX option.
Step 4: Update the filename and any parameters you want to change. This does not impact your ANT execution settings or start running any tests.
Note: You can either create a file from here or use the clipboard option to make a copy and paste it into your own property file in Visual Studio Code as you prefer.
Step 5: Using Visual Studio Code, you can now open this property file in your Provar projects’ ProvarDX™ subfolder. If you are using a VCS integration, you may want to add this folder to your repository to share it with others.
Creating this file from Provar Desktop is recommended as it prepopulates the required environment settings and parameters and is ready to run. There are some properties we want to briefly focus on here as they are extremely useful:
environmentlets you select a predefined Provar environment. This is important if you want to leverage the QA team’s work to predefine each environment, the screen resolution, default browser and most important a suite of connection overrides.
connectionOverridelets you provide name value pairs of a Provar Salesforce connection and the username you want to use instead during metadata or
runtestscommand execution. This is currently how you change your tests to run against a new temporary scratch org. Please refer to the Overriding Salesforce connections and tests section for an example of this in more detail.
testCaselets you change the list of tests (including wildcards) that are executed using runtests. This is currently how you can change the tests to be run.
testPlanlets you do the same as testCase but allows you to name one or more Provar Test Plans and is more powerful in terms of test case reuse. This requires your QA team to have already predefined Test Plans that you can make use of.
Step 6: Validate the property file especially if you have made any changes. You can do this using the Command Palette option, ProvarDX: Validate Properties File, or from the terminal window type.
sfdx provar:validate -p provardx-properties.json
Note: The provardx-properties.json filename is a default, like build.xml is with ANT. If you do not specify the property file, this special file will be selected by default if it is available in the local directory.
Step 7: You should now see the Property file is valid displayed in the output.
Whenever you want to execute these tests, you have two options.
From the terminal command line type:
sfdx provar:runtests -p provardx-properties.json
Or from the Visual Studio Code command palette, select the ProvarDX: Run Testcases option. You will be prompted for your property file, you can accept the default of provardx-properties.json.
Overriding Salesforce connections and tests
We can use the ProvarDX™ property file to pass a user substitution to run this same set of tests against any org with the same, or similar configuration. We can do this using the overrideConnections list in the provardx-properties.json file.
Step 1: Check the Salesforce orgs.
Step 2: The example above is a scratch org, (username [email protected]). This reflects configuration changes or development you may have just made to deliver a new feature to your Salesforce users.
If we wanted to run our tests exactly as defined by our Provar project we could simply use
sfdx provar:runtests -p provardx-properties.json
This is great if you want to run the same tests that your QA team already built in Provar, but it is not recommended for running those tests within your scratch org that you just made changes in.
To run the tests against your scratch org, you need to edit the provardx-properties.json file and add some lines to tell provardx to override the connections in the provardx project with our scratch org user. You can do this using the connectionOverride.
This takes the name of the connection in your Provar project. e.g. Admin in the case below, and reassigns all tests using that connection to use the scratch org instead.
Step 3: Now you can take those tests that you were working on in your admin connection and run them against this scratch org by re-running this command:
sfdx provar:runtests -p provardx-properties.json
This time you will see the original connection username ([email protected] in my case) replaced with our scratch org user. You can do this for multiple user connections, just make sure that you have added them to your DevHub first.
Step 4: To change the test cases that are executed, you can edit your provardx-properties to make the following changes:
- Amend the testCase JSON entry to change the test cases to execute. You can use a wildcard character of * to select multiple (e.g. “testCase” : [“/Sales/Acct Management 01.testcase”, “Service/*.testcase”]. This would run all of your tests in the service folder after running the Sales Acct management 01 test case.
- Add a plan entry to execute one or more Provar Test Plans: e.g.
This will execute both the DevSmoketest plan and the additional test cases.
ProvarDX™ is available as a beta feature and fully supported. However, there are still some known limitations with its usage and we’re planning further development to optimize the feature and address these issues. We reserve the right to change or remove features as needed to resolve these limitations and extend the product capability.
For a full list of up-to-date commands and parameters, take a look at the ProvarDX™ npm project listing.
If you’re new to SalesforceDX and the Salesforce CLI in general, take a look at the following resources from Salesforce:
- ProvarDX™ Installation
- SalesforceDX Integration using ANT (ConfigureDX)
- Trailhead: Quick Start SalesforceDX
- Trailhead: Developer Tools Installation
- Trailhead: App Development with SalesforceDX
- Trailhead Application Lifecycle Model
- ProvarDX™ npm listing and additional help can be accessed here.
- The ProvarDX™ Visual Studio Code extension can be accessed here
- General information
- Licensing Provar
- Provar trial guide and extensions
- Using Provar
- API testing
- Behavior-driven development
- Creating and importing projects
- Creating test cases
- Custom table mapping
- Debugging tests
- Defining a namespace prefix on a connection
- Defining proxy settings
- Environment management
- Exporting test cases into a PDF
- Exporting test projects
- Managing test steps
- Namespace org testing
- Provar desktop
- Provar Test Builder
- Refresh and Recompile
- Reload Org Cache
- Running tests
- Searching Provar with find usages
- Secrets management and encryption
- Setup and teardown test cases
- Tags and Service Level Agreements (SLAs)
- Test cycles
- Test plans
- Testing browser options
- Tooltip testing
- Using the Test Palette
- Test Palette introduction
- Control test steps
- List compare
- Read test step
- String test steps
- UI Test Steps
- Using custom APIs
- Callable tests
- Data-driven testing
- Page objects
- Block locator strategies
- Introduction to XPaths
- Creating an XPath
- Label locator strategies
- Maintaining page objects
- Mapping non-Salesforce fields
- Page object operations
- Refresh and reselect field locators in Test Builder
- Using Java method annotations for custom objects
- Applications testing
- Database testing
- Document testing
- Email testing
- Mobile testing
- OrchestraCMS Testing
- Salesforce CPQ testing
- ServiceMax testing
- Skuid testing
- Vlocity testing
- Webservices testing
- Introduction to test scheduling
- Apache Ant
- Configuration for sending emails via the Provar Command Line Interface (CLI)
- Continuous integration
- Azure DevOps
- Running a Provar CI task in Azure DevOps
- Configuring the Provar secrets password in Microsoft Azure Pipelines
- Parallel execution in Microsoft Azure Pipelines using multiple build.xml files
- Parallel execution in Microsoft Azure Pipelines using targets
- Parallel execution in Microsoft Azure Pipelines using Test Plans
- Bitbucket Pipelines
- GitHub Actions
- Running a Provar CI task in GitHub Actions
- Remote Trigger in GitHub Actions
- Parameterization using Environment Variables in GitHub Actions
- Parallel Execution in GitHub Actions using multiple build.xml files
- Parallel Execution in GitHub Actions using Targets
- Parallel Execution in GitHub Actions using Test Plan
- Parallel Execution in GitHub Actions using Job Matrix
- GitLab CI
- Travis CI
- Parallel Execution
- Running Provar on Linux
- Salesforce DX
- Team foundation server
- Version control
- Zephyr Cloud and Server
- Salesforce testing
- Adding a Salesforce connection
- Assert Page Error Messages on Add/Edit Product
- Dynamic Forms
- Internationalization support
- List and table testing
- Salesforce Release Updates
- Salesforce Lightning Testing
- Salesforce Lightning Web Component (LWC) locator support
- Salesforce console testing
- Visualforce Testing
- Testing best practices
- Configurations and permissions
- Error messages
- Licensing, installation and firewalls
- Test Builder and test cases
- Release notes