Integrating with Provar Manager

Provar Automation’s integration with Provar Manager is seamless and provides a great user experience. A brief overview of this integration is given below –

  • Provar Manager Connection – to create a connection to a Provar Manager instance.
  • Test Project Context Menu – to provide an option to upload the relevant Provar test project files to Provar Manager and associating Provar test files at the Test Plan level.
  • Test Plan configuration – to provide an option to report Test Plan execution to the Provar Manager.
  • Test Execution configuration – to provide an option to report test execution results to the Provar Manager.
  • Test Execution Results – The test execution results for Provar Manager reporting only. 

Setting up a Connection to Provar Manager

The Provar Manager connection is used to establish a connection with a Provar Manager instance. It is configured by creating a Provar Manager connection via the Connection tab. The Provar Manager connection (leveraging a normal Salesforce Connection) will only require a user login via OAuth Web Flow.

Prerequisites

There are some prerequisites for setting up a connection to Provar Manager as listed below:

  • A Salesforce org with Provar Manager must be already installed.
  • A Salesforce user with a Provar Manager license assigned, and the Provar Manager permission set assigned too.

How to set up an OAuth Web Flow connection to Provar Manager

Step 1: Create a Connected App

Step 2: Create a Provar Manager Connection (OAuth WebFlow).

In the Test Settings, navigate to the Connections tab and click plus (+) sign. The Add New Connection screen is displayed. Give a Connection Name and create an OAuth WebFlow Connection. In the Connection Type field, select Provar Manager then select Provar Manager Connection

For more information, please see Salesforce OAuth (Web Flow) connection

In the created OAuth WebFlow connection; you have to provide the details of the Consumer Key, Consumer Secret, Callback URL that are a part of  OAuth WebFlow connection.

Note: The above 3 fields are not the credentials of the user but they are the credentials of the Connected App.

Click Authorize and it will ask for your credentials; provide your credentials and when the authorization is successful, the Access Token and Refresh Token are populated.

Note: 

  • Provar Manager is only available via the Salesforce Lightning Experience.
  • Only one Provar Manager connection can be set.

Above: Snapshot of creating a Provar Manager Connection.

When the Provar Manager connection is created, you can see it under the Connections tab.

Above: Screenshot of Provar Manager Connection in Provar.

Creating and linking the Test Project in Provar Manager

Once you have created the Provar Manager connection; a project is created in Provar Manager with a Test Project Name and a unique Test Project Key. For example, the project AY Test Project is created in Provar Manager that has a unique key.

If a user wants to link a Provar test project with an existing Provar Manager project then he has to provide that key (unique key) while creating the Provar Manager connection in Provar. 

Note: The project created in Provar Manager with the same name and unique key.

In Provar Manager

Log on to the Provar Manager. Click Environments > Test Projects. You can see the AY Test Project created in Test Manager.

Above: Snapshot of the Project created in Provar Manager.

Click on the project (AY Test Project) and you can see the details of the project.

When you click on the project, you can see Test Plans inside it. When you click on Test Plans, you can see Test Suites (these are Test Plan Folders in Provar) and Test Cycles inside it.

Above: Details of the project created in Provar Manager.

Uploading screenshots

To upload screenshots, we have provided one option at connection level.  You can now select a Send Screenshots checkbox at the connection dialog box to upload screenshots.

This option is now available. If you have selected this checkbox then whatever screenshots are captured, they will all be uploaded. If its unchecked or de-selected, then no screenshot will be uploaded.

Note: The screenshot capture is allowed only in UI API’s. For example, in the UI Assert test case in the above test case example, you can see the options:

  • screenshot before
  • screenshot after

Note: In case of an error, a screenshot will always be created and uploaded. So, the maximum possible screenshots on a test step execution are 3 (before, after and on-error). And, the screenshots can be identified as (before_ and after_).

Removing a Provar Manager Connection

Removing or deleting a Provar Manager connection removes all the Provar Manager association. Right-click on Provar Manager connection to delete and the Provar Manager connection is deleted and the context-menu Upload to PM won’t be visible in Provar.

Object Mapping between Provar Automation and Provar Manager

The supported Provar test objects are mapped to the Provar Manager objects as shown below.

Above: Diagrammatic view of object mapping between Provar Automation and Provar Manager.

In Provar, the structure of Test Object(s) is organized as: Test Plan > Test Plan Folder > Test Instance > Test Case. If you click on a Test Instance, there is a test case inside it which is Test Case 1.

Above: Screenshot of Test Objects in Provar.

In the screenshot above, you can see AY Plan 1 > AY Folder 1 > Test Case 1.testinstance > Test Case 1. Testcase. All these are linked in hierarchy.

How to upload Test Plans, Test Plan Folders, Test Plan Instances, and Test Cases in Provar Manager

Prerequisite: 

To work with Provar Manager integration, users must have Test Plans (Test Plan Folders > Test Instance > Test Case) created before uploading or reporting execution to the Provar Manager.

In this section you will know how to upload Test Plans, Test Plan Folders, Test Plan Instances, and Test Cases in Provar Manager and you will also know about the parent-child hierarchy.

To upload a Test Plan from Provar to Provar Manager, right-click on the Test Plan (i.e. the test object) and click Upload to PM

Note: When you upload a Test Plan in Provar Manager, all the corresponding test objects in the Test Plan (i.e. Test Plan Folder, Test Instance and Test Case) will also be uploaded to the Provar Manager. You can upload (create/update), link, unlink these as required. For more information, see the below section Understanding the Parent-Child Hierarchy.

Understanding the Parent-Child Hierarchy

There is a parent-child hierarchy in the test objects. You can take advantage of the tree structure. For example, if you want to upload the Test Plan then all the contents of the Test Plan are also uploaded. So, you just have to select the Test Plan and Provar will automatically select all the contents of the Test Plan, you don’t need to select the contents individually.

Above: Image showing the tree structure hierarchy of a Test Plan.

Note: If you select any single test object then Provar will automatically select its child as well as parent too. Provar automatically creates the hierarchy and reduces any chances of an error.

For example, if you just want to upload a single test case then you can select it and Provar will automatically select its parent as well as its child. So, even if you select AY Folder 2, Provar will automatically select AY Plan 1 and you can upload or update it in Provar Manager.

Above: Screenshot displaying the hierarchy. Selecting a test case and Provar automatically selects the parent and child test objects.

Note: Hierarchy is only for the Test Plans. Environments are independent of  hierarchy as well as of each other. So, you have to select the environments individually as they are not connected to each other. For example, DEV and UAT environments are both different and have to be selected individually only.

Above: Selecting DEV and UAT environments separately.

The Upload to Provar Manager dialog provides the users with the ability to associate Provar Test Files to Provar Manager as listed below:

  • Creating a new object in Provar Manager from a Provar Test Object via upload.
  • Linking an existing Provar Manager object to a Provar Test Object via the “link” action.
  • Unlinking an associated Provar Test Object from Test Manager via the “unlink“ action.

Additional usability capabilities are also available to filter and identify when Provar Test Objects are matched, not matched and could be inferred as a match based on the matching criteria.

Filters

Let’s first take a look at the filters (the check boxes) on the top-right of the screen. You can use these as per your requirement. Although, these filters are self-explanatory we have provided brief explanations as mentioned below:

  • All Files – It will show all the files (that are in Provar).
  • Missing Files – The files i.e. the test objects that are not available in Provar Manager.
  • Changed Files – The files i.e. the test objects that have been changed. (For example, renamed).
  • Failures – In case, if some failure has occurred. Failure can only happen when you have uploaded something and it hasn’t been uploaded and it failed.

We have given some examples below. We have selected All Files and it is displayed as given below:

Above: Snapshot of filtering DEV environment that is renamed to DEV-Updated.

As a user if you want to upload Provar Test Objects to the Provar Manager, right-click on the Test Plan (i.e. test object) and click Upload to PM. Please see the screenshot given below.

 Above: Snapshot of Upload to PM context menu.

An Upload to Provar Manager dialog is displayed. You will also notice a message Fetching Test Objects from Provar Manager.

 Above: View of Upload to PM dialog fetching test objects from Provar Manager.

Select All files check box at the top of the dialog box and the fetched data is displayed in the dialog. Users can select and upload the relevant files to the Provar Manager.

You can see 5 fields on the Upload to Provar Manager dialog. These are explained below.

  • Provar Test Object – This field shows the test objects in Provar. 
  • Test Manager Object –  This field shows the  test objects in Provar Manager. 
  • Match Type – 
    • Inferred : It is an exact match of the Provar test object in Test Manager; but it isn’t linked. It has the same configuration in Provar Manager. You can just link it. You need to explicitly link it. (There is no need to create the object, it is already created.) 
    • Blank : The Provar test object is not available in Provar Manager.
    • Linked : The same test object is available in Provar as well as in Provar Manager. And, it is linked.
  • Status – 
    • Missing : The missing status is shown for test objects that are missing in Provar Manager. For example, you created a Test Plan but didn’t upload it in Provar Manager yet, so it will show the Status as Missing.
    • “?” : Either something has changed in the test object or the Provar is not able to comprehend the change. This is basically an unknown status. 

For example, you renamed a test object but didn’t upload it in Provar Manager (i.e there is a mismatch in Provar and Provar Manager). You must update it accordingly. So, when you upload it, it will also be renamed in Provar Manager.

  • “-” : There is no change in the data. For example, AY Plan 1  is AY Plan 1.
  • Failed – If there is an error while uploading the test object then Status is displayed as Failed and on hovering over it, you can see the message. 
  • Actions
    • Link : Click link to associate the Provar test object to Provar Manager. 
    • Unlink : Click unlink to dissociate the Provar test object to Test. Manager (If it is linked by mistake. For example, you linked Plan 1 to Plan 2.)

Note: If the data in Provar Test Object and Provar Manager Object field matches as  one-to -one then it will be displayed, otherwise it will be blank. The other columns contain additional information which is helpful to the user.

The Provar Test Object field shows the local Provar data i.e. the test objects in Provar. The Provar Manager Object field shows the Provar Manager data i.e. the test objects in Test Manager. If this data matches one-to-one, then Provar will show this data, otherwise it is blank. And, then we have some additional information (Match Type, Status, Actions) which are helpful for the user. 

For example, suppose the user had earlier uploaded a plan (AY Plan1) and this plan exists with the same name in Test Manager (AY Plan1) then it is an exact match and the Match Type is shown as Inferred; it is a potential match and you can link it as displayed in the Actions column.

In the Actions field, click link. A Test Manager Object selection dialog is displayed.

Note: It will display information depending on which test object the user clicked link. For example, if a user clicked the link against the Test Plan (AY Plan1); the he can see all the Plans only.

Above: User clicks link in the Actions field to link the Test Plan.

In the Provar Manager Object selection dialog, users can see only the plans listed. 

Note: If the user would have clicked the link against the Test Plan Folder, then the Test Plan folder is Test Suite in Provar Manager; then only test suite(s) would have been displayed.

Similarly, if the user would have clicked the link against the test case, then only test cases would have been displayed.

Note: The test objects (Test Case, Test Instance, Test Plan, Test Folder, except the Environment.) can be uploaded in Provar Manager in a similar way. The test object against which the user clicks link, only that is linked, there isn’t any mismatch.

Now, create a new environment in Provar.

In the bottom section of the Provar, click Environments. Click ‘+’. An Add new Test  Environment screen is displayed. Enter the Environment Name and Description. For more information, please see  Provar Environment Management

Note: Similarly, you can create environments as per your need, like QA, UAT etc.

Click Upload to Provar Manager. The Upload to Provar Manager dialog is displayed.

Select All files checkbox. All the test objects are loaded into the environment. So, this basically fetches all the data. Like here we have 5 types of objects – Test Case, Test Instance, Test Plan, Test Folder, and Environment. This triggers 5 different queries to the Provar Manager.

Note: Test Environments are always displayed at the top and they have no hierarchy.

So, if the user wants to see the environments then it will trigger a query for the environment in Test Manager. In Test Manager, click Environments and all the environments are displayed.

Above: Snapshot of Environments in Provar Manager. 

Next, we have triggered a query on all the Test Plans and all the Test Plans are displayed.

Example 1: Explaining the DEV-Updated example.

If we look at the above screenshot for better understanding. Provar shows the environment as DEV – Updated. Test Manager shows the environment as DEV. So, if you see here, Match Type shows as Linked (this means that data was earlier linked in local at sometime (when it was in sync), but now it is renamed.) and Status shows “?” (and this is the reason it shows a question mark (?)).

Example 2: An example when there is no change.

Let’s see another example of AY Plan 1. It shows  Match Type as Linked and Status as “- “; which means that there has been no change.

Now, you have to upload DEV-Updated. Click Upload. So, it will update it and not create it.

In Provar Manager

Click Environments. You’ll see DEV-Updated in Provar Manager.

Above: Snapshot of Dev-Updated example in Provar Manager. 

Uploading Callable Test Cases in Provar Manager

You can upload the callable test cases, if there is a dependency of a test case that is being uploaded to the Provar Manager. Select the callable test case checkbox to upload the callable test case.

You can see the Callable test cases at the bottom of the tree view under the  Callable Test Cases section. You can expand/collapse the Callable Test Cases section.  You can link/unlink callable test cases like any regular test cases. When a test case is selected, its related callable test cases are automatically selected. There is no limit to the nesting of callable test steps (i.e. a callable test can call another callable test) however, the testing is limited up to 3 levels.

Steps to upload Callable Test Cases

  1. Right-click on a Test Plan and click Upload to Provar Manager
  2. When you select a Test Plan, then all its dependent test cases and the Callable Test Cases are automatically selected. Click Upload.

Note: You can also select individual callable test cases, but you cannot deselect the dependent callable test cases from a plan.

3. You will see a message when the test cases are uploaded successfully.

In Provar Manager

In the Test Steps section, open any uploaded Test case within the test steps you will see callable test cases uploaded.

Open the callable test case and select the Test Case to Call field, it points to the dependent test case.

Open the test case to view its test steps.

Uploading Test Steps in Provar Manager

Now with the test case upload, we are also uploading the test steps. Now, suppose you want to upload a test case. For example, if you are uploading a “Break_with_while” test case. 

Note: A warning is displayed ”Uploading a test case will overwrite all the test steps in Provar Manager”. If other users have already created the test steps manually in Provar Manager and if you are going to upload your test case, that’s where this warning message will be helpful.

Above: Snapshot of warning message at the top that test steps will be overwritten. 

Previously, whenever you uploaded a test case, only the information (Details, System Information etc. as shown below in the screenshot) was populated in Provar Manager.

The addition is in the Test Steps section. So, if you are uploading a test case now, then you will also see the Test Steps uploaded as well. So, in the Test Steps section, you can see these test steps.  So, whatever you upload is the latest data.

Note: Whatever configuration, whatever Test steps you are having in Provar Manager (in the Test Steps section , it doesn’t matter) as it will get overwritten by the configuration or the information you are having in Provar Automation. So whatever information that you are having in your local Provar Studio that is the source of truth. So, all this information will be uploaded to the Provar Manager. 

Above: Snapshot of Test Steps uploaded in Provar Manager.

And, you also have the option to edit the test steps in Provar Manager; just click Edit. You can manually edit,  add test step (click Add Step), delete test step And, you can save it. Click Save. A message “Changes saved!” is displayed.

Limitation to uploading Test Steps in Provar Manager

  • Firstly, earlier we used to send/ upload test objects in 200 batches like send plans 200, folders 200, instance 200 and also send 200 test cases. The only change now is that 200 test cases won’t be sent. Test cases will be sent one-by-one to upload in PM. So, now test cases would be sent one-by-one, first test case then second test case and third and so on and so forth; because of this now the performance is somewhat slower as compared to the earlier scenario when test cases were sent in batches.
  • Secondly, the API of Salesforce that we are using, that API has a limitation. So, the one call that you send, in that you can make only 500 changes at maximum. That means, in your test case at 500 changes can be done at maximum. For example, you made a very lengthy test case that has 1000 steps that will not upload, it will fail.

Here, when we are saying that the limit is 500, it doesn’t mean that to create it is 500, so when it is overwritten, it means it will also have to delete this data and then upload the new data. So, it means that this collective information(deletion, insertion and update) is 500. 

Note: So, the user shouldn’t assume that they can send 500 steps at time. The collective information(creation, updation , insertion and deletion ) is 500. That means, they can send 500 test step changes. For example, we have a test case that has 500 test steps and we try to upload it. So, the test case upload will fail and it will throw this error message as shown in the screenshot below.

Provar Manager File Decoration

How will users know if a Test Plan structure is linked in Test Manager? Provar uses Provar Manager File Decoration to provide a visual hint to the user, when the relevant Provar test objects/files are linked to the Provar Manager. So, just by a glance the user can see which Test objects are already uploaded and linked in Provar Manager. 

A reference is presented to the user to show the linked object in Provar Manager and to provide visual feedback to the user; the word Provar Manager is displayed next to the file in the format [AssociationType] e.g. [ProvarManager]. For an associated Test Plan the Provar Manager File Decoration is displayed as given the screenshot below.

Note: This visual reference [ProvarManager] is displayed for all test objects that are linked to the Provar Manager i.e. Environments, Test Plans, Test Plan Folders, Test Instances and Test cases.

Above: Provar Manager File Decoration is displayed for the Test Plans.

Test Execution Reporting

When configuring a new Test Plan, the user must select an option to report test execution results to the Provar Manager.

Prerequisites 

  • A Connection to the Provar Manager must already be created. 
  • Configure the Test Plan and  provide an option to report test execution results to the Provar Manager.

Note: You need to create a Person record (or test record) that will be used as a tester in reporting execution, to signify who has initiated the execution, who is the tester. This Person record must use the email as “[email protected]

When Provar reports execution to the Provar Manager, we fill the tester field for that execution. This tester field in Provar Manager accepts Person type objects. So, there has to be a Person object created before reporting execution otherwise the reporting will fail.

How to configure Test Plans for reporting in Provar Manager

There are two cases in which you can configure Test Plans for reporting in Provar Manager as explained below.

Create a new Test Plan. For more information on how to create a Test Plan, please see Test Plan or double-click on the Test Plan, if you already have a Test Plan created. 

Case 1: When you are creating a new Test Plan.

When you create a new Test Plan, configure it for reporting to the Provar Manager. 

In the New Provar Test Plan screen, in the Reporting section, select the Provar Manager (PM) check box.

Note: The Connection Name (PM) is the name given to the Provar Manager connection by the user.

Case 2: If you already have a Test Plan created.

Double-click on an existing Test Plan. The Test Plan Editor is displayed.  

Click “+” (i.e. Add a new feature in Test Plan). And, select the Provar Manager(PM) reporting option.

Note: Once this PM reporting is configured, next time when you run or execute the Provar Manager, whichever option you have configured for Test Plan that will work, for example, PDF reporting etc.

In the Test Plan Editor, in the Provar Manager section, select the Enable Provar Manager Reporting check box. When enabled, once a test execution is completed (passed/failed) for the Test Plan and the associated Test Files, the results are reported to the Provar Manager.

Note: You can clear the check box if you don’t want the reporting in Provar Manager. 

Note: If you have enabled the reporting, but you haven’t uploaded the Test Plan then reporting will not be done. So, the user must first upload the Test Plan to the Provar Manager.

What is supported right now

As on date, what Provar supports is listed below:

  • Test Plan Execution Failure – If the Test Plan execution fails, the Provar Manager’s Test Execution object will be updated to reflect the failure.
  • Test Plan Folder Execution – On selecting a Test Plan Folder for execution, the Provar Test Execution will identify the Test Plan hierarchy (levels of Test Plan Folders) to the root Provar Test Plan and execute the selected Test Plan Folders.
  • Test Plan Instance Execution – On selecting a Test Plan Instance(s) for execution, the Provar Test Execution engine will identify the Test Plan hierarchy (levels of Test Plan Folders) to the root Provar Test Plan and execute the select Test Plan Instances.

Note: Running tests from Test Builder and Debug modes is not supported by Provar.

Running tests from Provar

As of now for running tests from the Run option is only supported. Once you have created a Test Plan, you can execute it. Click on the Test Plan and click Run. The Test Plan is executed. 

Note: The Test Plan is executed and there is no reporting in Provar Manager at this stage.

Above: Snapshot of Test Plan executed successfully.

Run with ANT build

When exporting Run Under ANT for a Test Plan you can configure reporting to Provar Manager, so that test execution results are reported to Provar Manager.

When the user selects the option to Run With ANT for a Test Plan, the user is presented with the Export ANT build file dialog.  In the Export ANT build file dialog, on the Details tab; in the Features section, the Enable Provar Manager (Test) reporting checkbox is selected by default if the Provar Manager connection is configured otherwise it isn’t selected.

How results are uploaded to Provar Manager 

To understand how results are uploaded to Provar Manager, first take a look at the object mapping between Provar Studio and Provar Manager when executing a test and uploading the test execution results to Provar Manager.

To get the reports in Provar Manager, you will need to configure the Test Plan. For example, which test cases have passed, this type of information should be there in Provar Manager reports.

So, here are some pre-conditions :

  • The Test Plan must be uploaded in Provar Manager.
  • Enable the configuration in the Test Plan for reporting in Provar Manager. 

Note: If you have only uploaded the Test plan in Provar Manager, it will not work. You will have to enable configuration for reporting in Provar Manager.

All the test files of the Test Plan must be uploaded as you need to run the test case(s) , and if you have not uploaded the test case then the report of the test case won’t be there in Provar Manager. When you execute the test case, a report will be generated in Provar Manager. 

Just before the test instance in green, we report it. See the screenshot below.

Above: Snapshot of Test Plan execution.

When you run your test, the report generated in Provar Manager has two main parts: 

  • Test Cycle – Test Cycle has the configuration information only. For example: 

which was the machine, what was the OS, who executed it, which Browser, which Environment. All this information is in the Test Cycle.

  • Test Execution – This section has the actual information like which test case you executed, for how much time you executed, how much time it took for execution, what’s the status (pass or fail).

In Provar Manager, the execution will look as shown below. 

Click Test Projects > Test Cycles. If a new session starts, it will show all the details. Then, a new record would be created in the Test Cycle. 

For example, here we have clicked on AY Plan 1.

Test Cycle Number is auto generated in Provar Manager. Provar doesn’t give this number. You can see the information, such as Plan, Environment, Browser, Resolution, OS Name, OS version, which testing tool, testing tool version etc. So, Test Cycle will show just the configuration information but nowhere it mentions which test case is executed and whether that test case has passed or failed; so for that there is a different object which comes with Test Execution.

Let’s take a look at the complete cycle as an example: For example, a user executed 2 test cases, so for these 2 test cases 2 reports are generated i.e. for these 2 test cases, 2 test execution objects are created.

For example, we have opened the first report.

Note: We have also covered the Test Step execution level  i.e. each and every single test step execution is reported.

As of now, we have showcased the complete test case execution at one go. So, if a test case will have multiple test steps, then each test step execution will be showcased.

Now that the test steps are uploaded in Provar Manager, the results are also uploaded and the test step errors are mapped to the Comments field.

Click on the Related section, here you can see that the execution records are created.

Click on the first execution which is the latest execution and you can see all these test step execution reports in the Test Step Executions section. 

Now, suppose if you click and open one test step; you can see all the details whether it passed or failed, its name, how much time it took for execution, what information is there in the logs. If the test step fails, the failure message can be seen in the Comments section.

At the end of the Test Plan, we are reporting Test Step execution and you will see a success message at the end of complete execution.

Note: You can also see the callable test case execution reports.

In the Test Step Executions section, you can also see the callable test case execution reports.

Above: Snapshot of callable test cases execution report in Provar Manager.

Support reporting to Provar Manager during ProvarDX execution

If the option to report to Provar Manager is enabled in the ProvarDX JSON settings, ProvarDX will report the test executions to Provar Manager as expected.

  • Test executions (test cycles, test executions, etc) are reported by ProvarDX to Provar Manager when the option is enabled or missing in the ProvarDX JSON settings.
  • Test executions (test cycles, test executions, etc…) are NOT reported by ProvarDX to Provar Manager when the option is disabled in the ProvarDX JSON settings.

Known Limitation

  • As of now in one Test Plan, the user cannot upload more than 350 test cases.
  • On uploading all the plans in a project, the upload dialog is getting stuck intermittently and the user is unable to proceed further in the dialog.
Review Provar on G2
Documentation library

Other available resources

Looking for something different?

We use cookies to better understand how our website is used so we can tailor content for you. For more information about the different cookies we use please take a look at our Privacy Policy.

Scroll to Top