AutoRABIT Salesforce DevOps in Provar Test
Provar supports integration with a large number of generic CI/CD and Salesforce specialist release management tools and platforms. For those customers using AutoRABIT Salesforce DevOps, this support article will guide you through the process of setting up your instance of AutoRABIT to run Provar tests. At the end of this article, you will have set up a continuous integration job in AutoRABIT Salesforce that runs your Provar tests in a headless environment.
Prerequisites of AutoRABIT in Provar Test
- An AutoRABIT Saleforce Professional or Enterprise license
- A standard license does not include cross-browser support or AutoRABIT’s Test Automation Factory (TAF).
- A valid Provar execution only or floating license
- Provar execution license(s) is/are recommended for use in CI/CD pipelines.
- A Provar project with at least one test case
- A Version Control System (VCS) that is already set up
- For this guide, we will be using Git. However, both Provar and AutoRABIT also support Subversion (SVN) and Team Foundation Server (TFS).
- Access to a Salesforce org to test.
- We recommend using a Sandbox or Developer org for testing purposes.
- Read more about permissions required to test Salesforce from Provar here: https://www.provartesting.com/support/general-information/granting-org-permissions-to-provar/
Other Support Articles For Reference
Introduction to test scheduling
Architecture Overview
Application Layer
The application layer consists of two major elements:
- A workstation running on macOS or Windows with a Fixed or Floating license Provar Desktop installation.
- A Version Control System (VCS) setup to contain your Provar test project(s). This can be Git, Subversion (SVN), or TFS based.
Note: To learn more about integrating Provar with your version control system of choice, please refer to Version Control and Devops.
Once you have created your Provar tests, or imported an existing project, and have made your initial check in to your chosen VCS, the next phase is implementing the environment layer.
Environment Layer
The environment layer consists of one major element, your AutoRABIT instance with a Professional or Enterprise license.
Note: The setup/implementation of AutoRABIT will not be covered and must be managed/supported by your AutoRABIT representative/consultant.
Once you have access to the AutoRABIT platform and have configured it accordingly, you must then configure Salesforce to ensure it is reachable from AutoRABIT’s servers.
The different pieces within the AutoRABIT platform that you will use include:
- CI Servers for executing CI Jobs on the AutoRABIT platform
- Test agents for running Provar tests on the AutoRABIT platform
- VCS integration to pull Provar tests from the Git repository
- Access to the Salesforce org you are testing from AutoRABIT via CI server(s) and test agent(s)
Platform Layer
The platform layer consists of one major element, your Salesforce org and any other cloud based systems you include in your end-to-end test cases. You cannot use AutoRABIT to test applications behind your company firewall unless you wish to open up access, which we do not recommend.
Provar Project Setup
In order to ensure that your Provar tests can be run successfully in the AutoRABIT environment, you must first configure the project accordingly.
Provar Version Limitations
Firstly, at the time of testing, AutoRABIT is using Provar 2.1.0.07 for their integration. This means that you must first ensure that your test project and test cases can be run on any of the following versions of Provar: 2.1.0.06, 2.1.0.07, or 2.1.1.09. In order to test this, you can download any of these Provar versions from the Provar Success Portal.
Once you have installed this locally, you can open your test project and run your tests with this version of Provar. If the tests run successfully on your local compatible version of Provar, then they should also pass in the AutoRABIT environment.
AutoRABIT Configuration
This section will guide you through all of the steps required to configure your AutoRABIT instance to run Provar tests.
Included in this, we will discuss the following topics:
- Setting up plugins in AutoRABIT
- Setting up version control in AutoRABIT
- Registering your Salesforce org for testing
- Configuring a CI Job in AutoRABIT
- Scheduling a CI Job in AutoRABIT
- Reporting and test results
That being said, you will need to make sure that your AutoRABIT user has permissions to perform the following actions using AutoRABIT permission names.
- Admin (This can be setup by an admin user rather than add these permissions to other user)
- Account Info
- Version control repositories
- Salesforce org management
- CI jobs
- Create a CI job
- CI job list
- CI job results
- Test Automation Factory (TAF)
For this guide, we are using an admin account in AutoRABIT for our configuration. However, it is recommended that you create a role in AutoRABIT for building Provar jobs and configuring them. An example role has been created below for reference.
All of the admin permissions are for setup purposes only. Once the plugins, the Salesforce orgs, and the repositories have been added, these permissions are no longer necessary for the rest of the setup.
Setting up plugins in AutoRABIT
In order to build a CI job in AutoRABIT that can run Provar tests, we must first add the desired plugins to our account. Once you have logged in, navigate to the Admin sidebar and go to My Account.
Scroll down to the Plugins section of the Account information page. Depending on your testing requirements and setup, you may add additional plugins where applicable. For instance, if you need to run your regression tests on Firefox and/or IE rather than Chrome, check the corresponding checkbox(es) for the Browsers section. For now, AutoRABIT only supports testing on the following browsers (currently they do not support testing on Edge or Safari):
- Chrome
- Firefox
- IE
Note: If you are using SVN or TFS as your VCS, then add those plugins instead. The rest of the steps throughout this guide will refer to Git, however the same steps can be applied to using any VCS.
The minimum required plugins for building a CI Job to execute Provar tests are shown below:
When adding Provar as a test type, you will need to specify your license file. To add your execution (or floating, if you do not have an execution) license, perform the following steps:
Step 1: Navigate to your $USER_HOME/Provar/.licenses folder and open your LICENSE_FILE_NAME.properties file in Notepad (or other text editor).
This properties file is automatically created/updated whenever you add a license inside of the Provar application. It is not recommended to edit this file unless a Provar consultant has informed you to do so.
Step 2: Your license file should resemble this example file. (The X’s represent your floating license key.)
#Fri May 15 11:11:00 CDT 2020
licenseStatus=Activated
licenseType=Floating
licenseKey=XXXXX-XXXXX-XXXXX-XXXXX-XXXXX
lastOnlineAvailabilityCheckUtc=1589559060573
Step 3: Create a copy of this license file and name it something unique such as Execution.properties. This copy will be edited to reflect your execution license, rather than your floating license, if applicable.
Step 4: Replace the licenseKey value with your execution license key, if you have one.
If you do not have a valid Provar Execution License Key, then every CI Job you run in AutoRABIT that executes Provar tests will use one Floating License Key. This means that your CI Job(s) will always be using one of your floating licenses throughout the duration of the job.. Keep this in mind to better track license usage across your organization. It is always recommended to use your Execution License key for all CI/CD integrations with Provar.
Step 5: To open the Provar credentials dialog box, click the pencil icon on the plugins screen here:
Step 6: In the Provar credentials dialog box, give the credentials a name, such as ProvarExecution.
Step 7: Specify the license file you just created and select Upload.
Note: Even though you can upload and select the license file, it does not validate the license key yet. You will not know if the license you’ve uploaded is valid until you run a test CI Job in AutoRABIT.
Once you’ve added all of the above plugins, don’t forget to click Save (this is located within the top left of the Plugins section) before exiting the application.
Setting up Version Control in AutoRABIT
After you have added the Git plugin, you can then register your Git Repository to AutoRABIT. This guide assumes that you have already set up a repository and checked in your Provar Project there. You can read more information about installing the VCS plugin(s) in Provar and checking in to a repository here: Introduction to Git Integration
You can follow the AutoRABIT documentation on how to set up and configure version control repositories here: https://support.autorabit.com/portal/kb/articles/creating-repository-in-autorabit
If you are using GitHub, here is an example of what your repository should look like:
The default branch will show as Master, however the target branch can be changed in the CI job itself.
Once the repository is added, verify the connection in AutoRABIT by selecting Test Connection.
The end result should resemble the following.
If you’d like to register additional branches, please refer to the AutoRABIT documentation here: https://support.autorabit.com/portal/kb/articles/registering-existing-branch-in-autorabit
Note: Metadata folder path will be empty for Provar test project repositories, and will not cause any errors when registering. You will need to provide a value for Last Commit Date, which can be any value, but must be filled in as it is a required field. Then, you simply skip the validation step and register the branch.
Registering your Salesforce Org for Testing
Before we can configure a CI job to run Provar tests, we must register a Salesforce org to AutoRABIT as our target. You can follow these steps from AutoRABIT’s documentation to register your Salesforce orgs here: https://support.autorabit.com/portal/kb/articles/registration-of-a-new-salesforce-org-in-autorabit
For this guide, we have added a single org using standard authentication. The end result looks like this.
Ensure that you have tested the connection here from AutoRABIT and that it passes successfully.
For configuring your AutoRABIT for deployments, you may need to configure your Salesforce org mappings, user permissions, etc. However, for the purposes of this guide, we only need to register the org and test its connectivity. You can read more about viewing your registered Salesforce orgs here: https://support.autorabit.com/portal/kb/articles/viewing-saved-salesforce-org
Note: For the purposes of this guide, registering a Salesforce org is only necessary so that we can select it as a target org in the CI Job configuration. However, this does not mean that the Provar tests will run against this environment. In fact, Provar tests will be run against whatever org that you set in the Salesforce Connect step(s) in your tests, environment overrides being applied where applicable for Test Plans.
Configuring a CI Job in AutoRABIT Salesforce
Now that our plugins and our VC repos have been successfully registered, we can begin creating our CI job in AutoRABIT. You can follow these steps from AutoRABIT’s documentation to set up a CI Job to run tests: https://docs.autorabit.com/19.3/Run%20Test%20Automation%20Scripts_print.htm
We will guide you through the process here as well.
Note: The following steps assume that you are using Chrome and Git.
On the sidebar, select CI Jobs and then select Create CI Job in the top right corner. From this menu, select Run Test Automation Scripts.
Note: You can combine steps listed above in pipelines, but for the purposes of this guide, we will only be running Provar test cases, not performing any deployments or any other CI operations. This is typical of how you would set up a nightly regression test for a specific org environment.
Give your CI job an easily recognizable name and description (optional).
Next, expand the Tests section. From here, select Fetch Test Cases From Provar.
Note: If you do not see Provar or Chrome as an option, make sure you have properly configured your plugins in AutoRABIT. Refer to the section on Setting up plugins in AutoRABIT.
To configure the Tests section, follow the below steps:
Step 1: Select your preferred VCS as the repository type.
Step 2: Select the Repository we added previously in the dropdown.
Step 3: Select a target branch from the repository you want to use to run the Provar test cases.
- You must register each branch individually in the VC Repo’s section. If you don’t see the branch you would like to use, refer back to the Setting up Version Control in AutoRABIT section.
Step 4: Select the Test Cases Root Path (a.k.a. the relative to your .testproject file in your repository).
For example, the Github repository looks like this:
Where ProvarIntegrations is the name of the repository and ProvarDevOps is the name of the folder in the repo that points to the Provar test project (e.g. the location of .testproject file). Therefore, the test cases root path would be ProvarDevOps.
Step 5: Set the Test Cases Execution Path (a.k.a. the path of the tests in your project that you want to be executed as part of the CI job).
This path can also be set accordingly to target Provar test plans. For an overview of Provar Test Plans and how to utilize them, see the following article: https://www.provartesting.com/support/using-provar/provar-test-plans/
Note: Test Plans allow you to incorporate environment overrides, multiple browser configurations, and grouping of test suites for consolidated reporting into your testing, without the need of modifying and keeping track of numerous build files. In fact, since AutoRABIT Saleforce does not currently support the passing of a test environment (to make use of Provar’s environment overrides on connections and variables/etc.), Test Plans are the best way of maintaining that functionality in your CI/CD pipelines.
For example, if the Github repository looks like this:
Here there exists a single Test Plan named Run All Tests. You could execute that test plan by configuring the job as follows:
If you want to run all of your test plans, you can specify the following:
When integrating your Provar Tests into your CI/CD Pipelines, we always recommend that you utilize Test Plans in Provar to execute groups of tests under certain conditions and against various environments.
If you do not have any test plans set up or you want to execute your tests in another manner, you can also specify folders/test cases in the tests folder of your project.
For example, if the Github repository looks like this:
In this example, there are two folders in the tests folder, namely Sales and Smoke Tests.
If you want to run all of the test cases in the Sales folder, you can set this field to tests/Sales.
If you want to run a particular test case, then specify that test case’s full path from the tests folder.
For example, in order to run RandomNumberTest.testcase, you would need to set the Test Cases Execution Path to tests/Smoke Tests/RandomNumberTest.testcase.
If you want to run all of the tests in your tests folder, then leave this field blank or simply put tests.
Step 6: Lastly, you will need to select all of the browsers you want to run your test cases on. We’ve only added the Chrome browser plugin, so for now we will select the Chrome checkbox.
Once the Tests section has been filled out, you will need to choose a target Salesforce org from the list of orgs that have been registered to AutoRABIT. In this example, there is only one registered org.
Scheduling CI Jobs
You can configure your CI jobs in AutoRABIT to execute on a specified schedule. The options for scheduling are as follows:
- No Schedule
- Use this option if you only want the job to run via manual trigger
- Daily
- Fixed time each day (e.g. once a day at midnight)
- Fixed Internals (e.g. twice a day at midnight and noon, or once every 12 hours)
- Weekly
- Choose the day and time each week the CI Job will run (e.g. every Sunday at 5:00 AM)
CI jobs in AutoRABIT can only be executed in one of the following ways:
- Schedule (via the Schedule Daily/Weekly option for the CI Job)
- Build Now (manually triggered in AutoRABIT)
For this guide, we have set the schedule to No Schedule and left the More Setting section as the default. In this example, you will be triggering the CI job manually which we recommend you do first to test if your job is working as expected. Afterwards you can update the schedule if you want to make this a daily regression test job.
You can read more about the More Settings section here: https://support.autorabit.com/portal/kb/articles/ci-job-more-settings-rotation-policy
For this particular CI job, that is all we need to configure. Click Save in the top right before continuing.
Once the CI job is saved, you can view it by selecting the CI Job List on the CI Jobs tab in AutoRABIT.
By default, your CI Job will show as Activated, meaning it will execute on the schedule you have set for the job. You can always deactivate the CI job by selecting the Activate/De-Activate slider. This will disable the schedule that was set for the CI job.
Triggering the CI Job
To test this CI Job, first we will trigger it manually. From the CI Jobs tab, select CI Job Results. This tab displays the results of all of your CI Jobs. Additionally, this is where you will go to trigger the CI jobs manually.
Select your job’s name from the Select Job dropdown. You should see a screen similar to the following:
Once you have some CI job results, you can also trigger the jobs in the top right corner of this same menu.
Now, let’s trigger this job to run our Provar tests. Select Build Now. You can set a Build Label to identify the particular build. For this first build, we are going to set this to Initial Build.
You can also set all of the browsers you want this particular build to execute against, if you have multiple browser plugins registered in AutoRABIT. Here, we just have Chrome registered, so you can leave all of the defaults and select Trigger Build.
If all goes well, you should see the following a build confirmation message:
Additionally, for more information about the other capabilities of CI Jobs outside of running your Provar tests, see the following guide: https://support.autorabit.com/portal/kb/autorabit-documentation/software-release-4-2/user-guide/ci-jobs.
Reporting and Test Results in AutoRABIT Salesforce
Once the build has been queued, you can view the build log by clicking on the notepad icon.
Reporting and Test Results
Click refresh periodically in order to update the build log window and follow the tests being executed.
The entire build log can be downloaded from this window as well, by clicking on the Download icon here:
In order to download the complete test results, click on the Download icon on the CI Job Results page once the build completes.
This will prompt you to download the entire build report produced by Provar, which includes the HTML and PDF reports.
Here is a snapshot of the zipped report that Provar generates:
If you extract the Ant_Results_Zip.zip and then delete the zipped file itself, you can see all of the reports/results here:
AutoRABIT Salesforce will give you additional information about the status of your test cases on the Build Results screen. You can view the results by clicking on the Job Name from the CI Job Results page.
If you click the number of passed/failed under the Functional Tests section, you can view the status of each test case.
Note: The AutoRABIT CI Job status does not reflect the status of your Provar tests in the CI Job. In other words, a build can complete successfully with failed tests.