AutoRABIT Salesforce DevOps in Provar Test
Provar supports integration with generic CI/CD and Salesforce specialist release management tools and platforms. For those customers using AutoRABIT Salesforce DevOps, this support article will guide you through setting up your instance of AutoRABIT to run Provar tests. At the end of this article, you will have set up a continuous integration job in AutoRABIT Salesforce that runs your Provar tests in a headless environment.
Prerequisites of AutoRABIT in Provar Test
- An AutoRABIT Salesforce Professional or Enterprise license
- A standard license does not include cross-browser support or AutoRABIT’s Test Automation Factory (TAF).
- A valid Provar execution only or floating license
- Provar execution license(s) is/are recommended for use in CI/CD pipelines.
- A Provar project with at least one test case
- A Version Control System (VCS) that is already set up
- For this guide, we will be using Git. However, Provar and AutoRABIT support Subversion (SVN) and Team Foundation Server (TFS).
- Access to a Salesforce org to test.
- We recommend using a Sandbox or Developer org for testing purposes.
- Read more about permissions required to test Salesforce from Provar here: https://www.provartesting.com/support/general-information/granting-org-permissions-to-provar/
Other Support Articles For Reference
Introduction to testing scheduling
Architecture Overview
Application Layer
The application layer consists of two major elements:
- A workstation runs on macOS or Windows with a Fixed or Floating license Provar Desktop installation.
- A Version Control System (VCS) setup to contain your Provar test project(s). This can be Git, Subversion (SVN), or TFS based.
Note: To learn more about integrating Provar with your version control system, please refer to Version Control and DevOps.
Once you have created your Provar tests, imported an existing project, and made your initial check-in to your chosen VCS, the next phase is implementing the environment layer.
Environment Layer
The environment layer consists of one major element, your AutoRABIT instance with a Professional or Enterprise license.
Note: The setup/implementation of AutoRABIT will not be covered and must be managed/supported by your AutoRABIT representative/consultant.
Once you have access to the AutoRABIT platform and have configured it accordingly, you must configure Salesforce to ensure it is reachable from AutoRABIT’s servers.
The different pieces within the AutoRABIT platform that you will use include the following:
- CI Servers for executing CI Jobs on the AutoRABIT platform
- Test agents for running Provar tests on the AutoRABIT platform
- VCS integration to pull Provar tests from the Git repository
- Access to the Salesforce org you are testing from AutoRABIT via CI server(s) and test agent(s)
Platform Layer
The platform layer consists of one major element, your Salesforce org and any other cloud-based systems you include in your end-to-end test cases. You cannot use AutoRABIT to test applications behind your company firewall unless you wish to open up access, which we do not recommend.
Provar Project Setup
To ensure that your Provar tests can be run successfully in the AutoRABIT environment, you must first configure the project accordingly.
Provar Version Limitations
Firstly, at the time of testing, AutoRABIT is using Provar 2.1.0.07 for their integration. This means that you must first ensure that your test project and test cases can be run on any of the following versions of Provar: 2.1.0.06, 2.1.0.07, or 2.1.1.09. To test this, you can download any Provar versions from the Provar Success Portal.
Once you have installed this locally, you can open your test project and run your tests with this version of Provar. If the tests run successfully on your local compatible version of Provar, then they should also pass in the AutoRABIT environment.
AutoRABIT Configuration
This section will guide you through all the steps required to configure your AutoRABIT instance to run Provar tests.
Included in this, we will discuss the following topics:
- Setting up plugins in AutoRABIT
- Setting up version control in AutoRABIT
- Registering your Salesforce org for testing
- Configuring a CI Job in AutoRABIT
- Scheduling a CI Job in AutoRABIT
- Reporting and test results
That being said, you must ensure that your AutoRABIT user has permission to perform the following actions using AutoRABIT permission names.
- Admin (This can be set up by an admin user rather than adding these permissions to other users)
- Account Info
- Version control repositories
- Salesforce org management
- CI jobs
- Create a CI job
- CI job list
- CI job results
- Test Automation Factory (TAF)
For this guide, we are using an admin account in AutoRABIT for our configuration. However, it is recommended that you create a role in AutoRABIT for building Provar jobs and configuring them. An example role has been created below for reference.
All of the admin permissions are for setup purposes only. Once the plugins, the Salesforce orgs, and the repositories have been added, these permissions are no longer necessary for the rest of the setup.
Setting up plugins in AutoRABIT
We must first add the desired plugins to our account to build a CI job in AutoRABIT that can run Provar tests. Once you have logged in, please navigate to the Admin sidebar and go to My Account.
Scroll down to the Plugins section of the Account information page. Depending on your testing requirements and setup, you may add additional plugins where applicable. For instance, if you need to run your regression tests on Firefox and/or IE rather than Chrome, check the corresponding checkbox(es) for the Browsers section. For now, AutoRABIT only supports testing on the following browsers (currently, they do not support testing on Edge or Safari):
- Chrome
- Firefox
- IE
Note: If you use SVN or TFS as your VCS, add those plugins instead. The rest of the steps throughout this guide will refer to Git. However, the same steps can be applied to using any VCS.
The minimum required plugins for building a CI Job to execute Provar tests are shown below:
You must specify your license file when adding Provar as a test type. To add your execution (or floating, if you do not have an execution) license, perform the following steps:
Step 1: Navigate to your $USER_HOME/Provar/.licenses folder and open your LICENSE_FILE_NAME.properties file in Notepad (or other text editors).
This properties file is automatically created/updated whenever you add a license inside the Provar application. It is not recommended to edit this file unless a Provar consultant has informed you to do so.
Step 2: Your license file should resemble this example file. (The Xs represent your floating license key.)
#Fri May 15 11:11:00 CDT 2020
licenseStatus=Activated
licenseType=Floating
licenseKey=XXXXX-XXXXX-XXXXX-XXXXX-XXXXX
lastOnlineAvailabilityCheckUtc=1589559060573
Step 3: Create a copy of this license file and name it something unique such as Execution.properties. This copy will be edited to reflect your execution license rather than your floating license, if applicable.
Step 4: Replace the license-key value with your execution license key if you have one.
If you do not have a valid Provar Execution License Key, then every CI Job you run in AutoRABIT that executes Provar tests will use one Floating License Key. This means your CI Job(s) will always use one of your floating licenses throughout the job. Keep this in mind to better track license usage across your organization. It is always recommended to use your Execution License key for all CI/CD integrations with Provar.
Step 5: To open the Provar credentials dialog box, click the pencil icon on the plugins screen here:
Step 6: In the Provar credentials dialog box, give the credentials a name, such as ProvarExecution.
Step 7: Specify the license file you just created and select Upload.
Note: Although you can upload and select the license file, it does not validate the license key yet. You will not know if your uploaded license is valid until you run a test CI Job in AutoRABIT.
Once you’ve added all of the above plugins, don’t forget to click Save (located within the top left of the Plugins section) before exiting the application.
Setting up Version Control in AutoRABIT
After adding the Git plugin, you can register your Git Repository to AutoRABIT. This guide assumes that you have already set up a repository and checked in your Provar Project there. You can read more information about installing the VCS plugin(s) in Provar and checking in to a repository here: Introduction to Git Integration.
You can follow the AutoRABIT documentation on how to set up and configure version control repositories here: https://support.autorabit.com/portal/kb/articles/creating-repository-in-autorabit.
If you are using GitHub, here is an example of what your repository should look like:
The default branch will show as Master. However, the target branch can be changed in the CI job itself.
Once the repository is added, verify the connection in AutoRABIT by selecting Test Connection.
The result should resemble the following.
If you’d like to register additional branches, please refer to the AutoRABIT documentation here: https://support.autorabit.com/portal/kb/articles/registering-existing-branch-in-autorabit.
Note: The metadata folder path will be empty for Provar test project repositories and will not cause errors when registering. You must provide a value for Last Commit Date, which can be any value but must be filled in as it is a required field. Then, you skip the validation step and register the branch.
Registering your Salesforce Org for Testing
Before configuring a CI job to run Provar tests, we must register a Salesforce org to AutoRABIT as our target. You can follow these steps from AutoRABIT’s documentation to register your Salesforce orgs here: https://support.autorabit.com/portal/kb/articles/registration-of-a-new-salesforce-org-in-autorabit.
For this guide, we have added a single org using standard authentication. The result looks like this.
Ensure that you have tested the connection here from AutoRABIT and that it passes successfully.
For configuring your AutoRABIT for deployments, you may need to configure your Salesforce org mappings, user permissions, etc. However, we only need to register the org and test its connectivity for this guide. You can read more about viewing your registered Salesforce orgs here: https://support.autorabit.com/portal/kb/articles/viewing-saved-salesforce-org.
Note: For this guide, registering a Salesforce org is only necessary so that we can select it as a target org in the CI Job configuration. However, this does not mean the Provar tests will run against this environment. Provar tests will be run against whatever org you set in the Salesforce Connect step(s) in your tests, with environment overrides being applied where applicable for Test Plans.
Configuring a CI Job in AutoRABIT Salesforce
Now that our plugins and VC repost have been registered, we can begin creating our CI job in AutoRABIT. You can follow these steps from AutoRABIT’s documentation to set up a CI Job to run tests: https://docs.autorabit.com/19.3/Run%20Test%20Automation%20Scripts_print.htm.
We will guide you through the process here as well.
Note: The following steps assume that you are using Chrome and Git.
Select CI Jobs and Create CI Job in the top right corner of the sidebar. From this menu, choose Run Test Automation Scripts.
Note: You can combine the steps listed above in pipelines, but for this guide, we will only be running Provar test cases, not performing any deployments or any other CI operations. This is typical of how you would set up a nightly regression test for a specific org environment.
Give your CI job an easily recognizable name and description (optional).
Next, expand the Tests section. From here, select Fetch Test Cases From Provar.
Note: If you do not see Provar or Chrome as an option, ensure you have correctly configured your plugins in AutoRABIT. Refer to the section on Setting up plugins in AutoRABIT.
To configure the Tests section, follow the below steps:
Step 1: Select your preferred VCS as the repository type.
Step 2: Select the Repository we added previously in the dropdown.
Step 3: Select a target branch from the repository you want to use to run the Provar test cases.
- You must register each branch individually in the VC Repo’s section. If you don’t see the branch you want to use, refer to the Setting up Version Control in the AutoRABIT section.
Step 4: Select the Test Cases Root Path (a.k.a. the relative to your .testproject file in your repository).
For example, the Github repository looks like this:
ProvarIntegrations is the repository’s name, and ProvarDevOps is the folder’s name in the repo that points to the Provar test project (e.g., the location of the .testproject file). Therefore, the test case’s root path would be ProvarDevOps.
Step 5: Set the Test Cases Execution Path (a.k.a. the path of the tests in your project that you want to be executed as part of the CI job).
This path can also be set accordingly to target Provar test plans. For an overview of Provar Test Plans and how to utilize them, see the following article: https://www.provartesting.com/support/using-provar/provar-test-plans/
Note: Test Plans allow you to incorporate environment overrides, multiple browser configurations, and grouping test suites for consolidated reporting into your testing without needing to modify and keep track of multiple build files. Since AutoRABIT Salesforce does not currently support the passing of a test environment (to make use of Provar’s environment overrides on connections and variables/etc.), Test Plans are the best way of maintaining that functionality in your CI/CD pipelines.
For example, if the GitHub repository looks like this:
Here there exists a single Test Plan named Run All Tests. You could execute that test plan by configuring the job as follows:
If you want to run all of your test plans, you can specify the following:
When integrating your Provar Tests into your CI/CD Pipelines, we recommend you utilize Test Plans in Provar to execute groups of tests under certain conditions and against various environments.
If you do not have any test plans set up or want to execute your tests in another manner, you can also specify folders/test cases in the tests folder of your project.
For example, if the GitHub repository looks like this:
In this example, two folders are in the tests folder: Sales and Smoke Tests.
If you want to run all the test cases in the Sales folder, you can set this field to tests/Sales.
If you want to run a particular test case, then specify that test case’s entire path from the tests folder.
For example, to run RandomNumberTest.testcase, you must set the Test Cases Execution Path to tests/Smoke Tests/RandomNumberTest.testcase.
If you want to run all the tests in your tests folder, leave this field blank or put tests.
Step 6: Lastly, you must select all the browsers you want to run your test cases on. We’ve only added the Chrome browser plugin, so we will choose the Chrome checkbox for now.
Once the Tests section has been filled out, you must choose a target Salesforce org from the list of orgs registered to AutoRABIT. In this example, there is only one registered org.
Scheduling CI Jobs
You can configure your CI jobs in AutoRABIT to execute on a specified schedule. The options for scheduling are as follows:
- No Schedule
- Use this option if you only want the job to run via a manual trigger
- Daily
- Fixed time each day (e.g., once a day at midnight)
- Fixed Internals (e.g., twice a day at midnight and noon or once every 12 hours)
- Weekly
- Choose the day and time each week the CI Job will run (e.g., every Sunday at 5:00 AM)
CI jobs in AutoRABIT can only be executed in one of the following ways:
- Schedule (via the Schedule Daily/Weekly option for the CI Job)
- Build Now (manually triggered in AutoRABIT)
For this guide, we have set the schedule to No Schedule and left the More Setting section as the default. In this example, you will trigger the CI job manually, which we recommend you do first to test if your job is working as expected. Afterward, you can update the schedule to make this a daily regression test job.
You can read more about the More Settings section here: https://support.autorabit.com/portal/kb/articles/ci-job-more-settings-rotation-policy.
For this particular CI job, that is all we need to configure. Click Save in the top right before continuing.
Once the CI job is saved, you can view it by selecting the CI Job List on the CI Jobs tab in AutoRABIT.
By default, your CI Job will show as Activated, meaning it will execute on the schedule you have set for the job. You can permanently deactivate the CI job by selecting the Activate/De-Activate slider. This will disable the program that was set for the CI job.
Triggering the CI Job
To test this CI Job, first, we will trigger it manually. From the CI Jobs tab, select CI Job Results. This tab displays the results of all of your CI Jobs. Additionally, this is where you will go to trigger the CI jobs manually.
Select your job’s name from the job select Job dropdown. You should see a screen similar to the following:
Once you have some CI job results, you can trigger the jobs in the top right corner of this menu.
Now, let’s trigger this job to lets Provar tests. Select Build Now. You can set a Build Label to identify the particular build. We will set this to Initial Build for this first build.
You can also set all browsers you want this particular build to execute against if you have multiple browser plugins registered in AutoRABIT. We have Chrome registered here, so you can leave all the defaults and select Trigger Build.
If all goes well, you should see the following build confirmation message:
Additionally, for more information about the other capabilities of CI Jobs outside of running your Provar tests, see the following guide: https://support.autorabit.com/portal/kb/autorabit-documentation/software-release-4-2/user-guide/ci-jobs.
Reporting and Test Results in AutoRABIT Salesforce
Once the build has been queued, you can view the build log by clicking on the notepad icon.
Reporting and Test Results
Click refresh periodically to update the build log window and follow the executed tests.
The entire build log can be downloaded from this window as well by clicking on the Download icon here:
Once the build completes, click the Download icon on the CI Job Results page to download the complete test results.
This will prompt you to download the entire build report produced by Provar, which includes the HTML and PDF reports.
Here is a snapshot of the zipped report that Provar generates:
If you extract the Ant_Results_Zip.zip and then delete the zipped file itself, you can see all of the reports/results here:
AutoRABIT Salesforce will give you additional information about the status of your test cases on the Build Results screen. You can view the results by clicking the Job Name from the CI Job Results page.
If you click the number of passed/failed under the Functional Tests section, you can view the status of each test case.
Note: The AutoRABIT CI Job-status does not reflect the status of your Provar tests in the CI Job. In other words, a build can complete successfully with failed tests.