Integrating with Test Manager
Provar’s integration with *Test Manager is seamless and provides a great user experience. A brief overview of this integration is given below –
- Test Manager Connection – to create a connection to a Test Manager instance.
- Test Project Context Menu – to provide an option to upload the relevant Provar test project files to Test Manager and associating Provar test files at the Test Plan level.
- Test Plan configuration – to provide an option to report Test Plan execution to the Test Manager.
- Test Execution configuration – to provide an option to report test execution results to the Test Manager.
- Test Execution Results – The test execution results for Test Manager reporting only.
*Test Manager now has a new name of Provar Manager, to reflect the wider scope of quality management and analysis. This will be reflected in future product changes and documentation updates.
Setting up a Connection to Test Manager
The Test Manager connection is used to establish a connection with a Test Manager instance. It is configured by creating a Test Manager connection via the Connection tab. The Test Manager connection (leveraging a normal Salesforce Connection) will only require a user login via OAuth Web Flow.
There are some prerequisites for setting up a connection to Test Manager as listed below:
- A Salesforce org with Test Manager must be already installed.
- A Salesforce user with a Test Manager license assigned, and the Test Manager permission set assigned too.
How to set up an OAuth Web Flow connection to Test Manager
Step 1: Create a Connected App.
Step 2: Create a Test Manager Connection (OAuth WebFlow).
In the Test Settings, navigate to the Connections tab and click plus (+) sign. The Add New Connection screen is displayed. Give a Connection Name and create an OAuth WebFlow Connection. In the Connection Type field, select Test Manager then select Test Manager Connection.
For more information, please see Salesforce OAuth (Web Flow) connection.
In the created OAuth WebFlow connection; you have to provide the details of the Consumer Key, Consumer Secret, Callback URL that are a part of OAuth WebFlow connection.
Note: The above 3 fields are not the credentials of the user but they are the credentials of the Connected App.
Click Authorize and it will ask for your credentials; provide your credentials and when the authorization is successful, the Access Token and Refresh Token are populated.
- Test Manager is only available via the Salesforce Lightning Experience.
- Only one Test Manager connection can be set.
Above: Snapshot of creating a Test Manager Connection.
When the Test Manager connection is created, you can see it under the Connections tab.
Above: Screenshot of Test Manager Connection in Provar.
Creating and linking the Test Project in Test Manager
Once you have created the Test Manager connection; a project is created in Test Manager with a Test Project Name and a unique Test Project Key. For example, the project AY Test Project is created in Test Manager that has a unique key.
If a user wants to link a Provar test project with an existing Test Manager project then he has to provide that key (unique key) while creating the Test Manager connection in Provar.
Note: The project created in Test Manager with the same name and unique key.
In Test Manager –
Log on to the Test Manager. Click Environments > Test Projects. You can see the AY Test Project created in Test Manager.
Above: Snapshot of the Project created in Test Manager.
Click on the project (AY Test Project) and you can see the details of the project.
When you click on the project, you can see Test Plans inside it. When you click on Test Plans, you can see Test Suites (these are Test Plan Folders in Provar) and Test Cycles inside it.
Above: Details of the project created in Test Manager.
Removing a Test Manager Connection
Removing or deleting a Test Manager connection removes all the Test Manager association. Right-click on the Test Manager connection to delete and the Test Manager connection is deleted and the context-menu Upload to TM won’t be visible in Provar.
Object Mapping between Provar Studio and Test Manager
The supported Provar test objects are mapped to the Test Manager objects as shown below.
Above: Diagrammatic view of object mapping between Provar Studio and Test Manager.
In Provar, the structure of Test Object(s) is organized as: Test Plan > Test Plan Folder > Test Instance > Test Case. If you click on a Test Instance, there is a test case inside it which is Test Case 1.
Above: Screenshot of Test Objects in Provar.
In the screenshot above, you can see AY Plan 1 > AY Folder 1 > Test Case 1.testinstance > Test Case 1. Testcase. All these are linked in hierarchy.
How to upload Test Plans, Test Plan Folders, Test Plan Instances, and Test Cases in Test Manager
To work with Test Manager integration, users must have Test Plans (Test Plan Folders > Test Instance > Test Case) created before uploading or reporting execution to the Test Manager.
In this section you will know how to upload Test Plans, Test Plan Folders, Test Plan Instances, and Test Cases in Test Manager and you will also know about the parent-child hierarchy.
To upload a Test Plan from Provar to Test Manager, right-click on the Test Plan (i.e. the test object) and click Upload to TM.
Note: When you upload a Test Plan in Test Manager, all the corresponding test objects in the Test Plan (i.e. Test Plan Folder, Test Instance and Test Case) will also be uploaded to the Test Manager. You can upload (create/update), link, unlink these as required. For more information, see the below section Understanding the Parent-Child Hierarchy.
Understanding the Parent-Child Hierarchy
There is a parent-child hierarchy in the test objects. You can take advantage of the tree structure. For example, if you want to upload the Test Plan then all the contents of the Test Plan are also uploaded. So, you just have to select the Test Plan and Provar will automatically select all the contents of the Test Plan, you don’t need to select the contents individually.
Above: Image showing the tree structure hierarchy of a Test Plan.
Note: If you select any single test object then Provar will automatically select its child as well as parent too. Provar automatically creates the hierarchy and reduces any chances of an error.
For example, if you just want to upload a single test case then you can select it and Provar will automatically select its parent as well as its child. So, even if you select AY Folder 2, Provar will automatically select AY Plan 1 and you can upload or update it in Test Manager.
Above: Screenshot displaying the hierarchy. Selecting a test case and Provar automatically selects the parent and child test objects.
Note: Hierarchy is only for the Test Plans. Environments are independent of hierarchy as well as of each other. So, you have to select the environments individually as they are not connected to each other. For example, DEV and UAT environments are both different and have to be selected individually only.
Above: Selecting DEV and UAT environments separately.
The Upload to Test Manager dialog provides the users with the ability to associate Provar Test Files to Test Manager as listed below:
- Creating a new object in Test Manager from a Provar Test Object via upload.
- Linking an existing Test Manager object to a Provar Test Object via the “link” action.
- Unlinking an associated Provar Test Object from Test Manager via the “unlink“ action.
Additional usability capabilities are also available to filter and identify when Provar Test Objects are matched, not matched and could be inferred as a match based on the matching criteria.
Let’s first take a look at the filters (the check boxes) on the top-right of the screen. You can use these as per your requirement. Although, these filters are self-explanatory we have provided brief explanations as mentioned below:
- All Files – It will show all the files (that are in Provar).
- Missing Files – The files i.e. the test objects that are not available in Test Manager.
- Changed Files – The files i.e. the test objects that have been changed. (For example, renamed).
- Failures – In case, if some failure has occurred. Failure can only happen when you have uploaded something and it hasn’t been uploaded and it failed.
We have given some examples below. We have selected All Files and it is displayed as given below:
Above: Snapshot of filtering DEV environment that is renamed to DEV-Updated.
As a user if you want to upload Provar Test Objects to the Test Manager, right-click on the Test Plan (i.e. test object) and click Upload to TM. Please see the screenshot given below.
Above: Snapshot of Upload to TM context menu.
An Upload to Test Manager dialog is displayed. You will also notice a message Fetching Test Objects from Test Manager.
Above: View of Upload to TM dialog fetching test objects from Test Manager.
Select All files check box at the top of the dialog box and the fetched data is displayed in the dialog. Users can select and upload the relevant files to the Test Manager.
You can see 5 fields on the Upload to Test Manager dialog. These are explained below.
- Provar Test Object – This field shows the test objects in Provar.
- Test Manager Object – This field shows the test objects in Test Manager.
- Match Type –
- Inferred : It is an exact match of the Provar test object in Test Manager; but it isn’t linked. It has the same configuration in Test Manager. You can just link it. You need to explicitly link it. (There is no need to create the object, it is already created.)
- Blank : The Provar test object is not available in Test Manager.
- Linked : The same test object is available in Provar as well as in Test Manager. And, it is linked.
- Status –
- Missing : The missing status is shown for test objects that are missing in Test Manager. For example, you created a Test Plan but didn’t upload it in Test Manager yet, so it will show the Status as Missing.
- “?” : Either something has changed in the test object or the Provar is not able to comprehend the change. This is basically an unknown status.
For example, you renamed a test object but didn’t upload it in Test Manager (i.e there is a mismatch in Provar and Test Manager). You must update it accordingly. So, when you upload it, it will also be renamed in Test Manager.
- “-” : There is no change in the data. For example, AY Plan 1 is AY Plan 1.
- Failed – If there is an error while uploading the test object then Status is displayed as Failed and on hovering over it, you can see the message.
- Actions –
- Link : Click link to associate the Provar test object to Test Manager.
- Unlink : Click unlink to dissociate the Provar test object to Test. Manager (If it is linked by mistake. For example, you linked Plan 1 to Plan 2.)
Note: If the data in Provar Test Object and Test Manager Object field matches as one-to -one then it will be displayed, otherwise it will be blank. The other columns contain additional information which is helpful to the user.
The Provar Test Object field shows the local Provar data i.e. the test objects in Provar. The Test Manager Object field shows the Test Manager data i.e. the test objects in Test Manager. If this data matches one-to-one, then Provar will show this data, otherwise it is blank. And, then we have some additional information (Match Type, Status, Actions) which are helpful for the user.
For example, suppose the user had earlier uploaded a plan (AY Plan1) and this plan exists with the same name in Test Manager (AY Plan1) then it is an exact match and the Match Type is shown as Inferred; it is a potential match and you can link it as displayed in the Actions column.
In the Actions field, click link. A Test Manager Object selection dialog is displayed.
Note: It will display information depending on which test object the user clicked link. For example, if a user clicked the link against the Test Plan (AY Plan1); the he can see all the Plans only.
Above: User clicks link in the Actions field to link the Test Plan.
In the Test Manager Object selection dialog, users can see only the plans listed.
Note: If the user would have clicked the link against the Test Plan Folder, then the Test Plan folder is Test Suite in Test Manager; then only test suite(s) would have been displayed as shown in the screenshot below.
Above: View of Test Manager Object Selection Dialog displaying Test Suites.
Similarly, if the user would have clicked the link against the test case, then only test cases would have been displayed.
Note: The test objects (Test Case, Test Instance, Test Plan, Test Folder, except the Environment.) can be uploaded in Test Manager in a similar way. The test object against which the user clicks link, only that is linked, there isn’t any mismatch.
Now, create a new environment in Provar.
In the bottom section of the Provar, click Environments. Click ‘+’. An Add new Test Environment screen is displayed. Enter the Environment Name and Description. For more information, please see Provar Environment Management
Note: Similarly, you can create environments as per your need, like QA, UAT etc.
Click Upload to Test Manager. The Upload to Test Manager dialog is displayed.
Select All files checkbox. All the test objects are loaded into the environment. So, this basically fetches all the data. Like here we have 5 types of objects – Test Case, Test Instance, Test Plan, Test Folder, and Environment. This triggers 5 different queries to the Test Manager.
Note: Test Environments are always displayed at the top and they have no hierarchy.
So, if the user wants to see the environments then it will trigger a query for the environment in Test Manager. In Test Manager, click Environments and all the environments are displayed.
Above: Snapshot of Environments in Test Manager.
Next, we have triggered a query on all the Test Plans and all the Test Plans are displayed.
Example 1: Explaining the DEV -Updated example.
If we look at the above screenshot for better understanding. Provar shows the environment as DEV – Updated. Test Manager shows the environment as DEV. So, if you see here, Match Type shows as Linked (this means that data was earlier linked in local at sometime (when it was in sync), but now it is renamed.) and Status shows “?” (and this is the reason it shows a question mark (?)).
Example 2: An example when there is no change.
Let’s see another example of AY Plan 1. It shows Match Type as Linked and Status as “- “; which means that there has been no change.
Now, you have to upload DEV-Updated. Click Upload. So, it will update it and not create it.
In Test Manager –
Click Environments. You’ll see DEV-Updated in Test Manager.
Above: Snapshot of Dev-Updated example in Test Manager.
Test Manager File Decoration
How will users know if a Test Plan structure is linked in Test Manager? Provar uses Test Manager File Decoration to provide a visual hint to the user, when the relevant Provar test objects/files are linked to the Test Manager. So, just by a glance the user can see which Test objects are already uploaded and linked in Test Manager.
A reference is presented to the user to show the linked object in Test Manager and to provide visual feedback to the user; the word Test Manager is displayed next to the file in the format [AssociationType] e.g. [TestManager]. For an associated Test Plan the Test Manager File Decoration is displayed as given the screenshot below.
Note: This visual reference [TestManager] is displayed for all test objects that are linked to the Test Manager i.e. Environments, Test Plans, Test Plan Folders, Test Instances and Test cases.
Above: Test Manager File Decoration is displayed for the Test Plans.
Test Execution Reporting
When configuring a new Test Plan, the user must select an option to report test execution results to the Test Manager.
- A Connection to the Test Manager must already be created.
- Configure the Test Plan and provide an option to report test execution results to the Test Manager.
Note: You need to create a Person record (or test record) that will be used as a tester in reporting execution, to signify who has initiated the execution, who is the tester. This Person record must use the email as “[email protected]”
When Provar reports execution to the Test Manager, we fill the tester field for that execution. This tester field in Test Manager accepts Person type objects. So, there has to be a Person object created before reporting execution otherwise the reporting will fail.
How to configure Test Plans for reporting in Test Manager
There are two cases in which you can configure Test Plans for reporting in Test Manager as explained below.
Create a new Test Plan. For more information on how to create a Test Plan, please see Test Plan or double-click on the Test Plan, if you already have a Test Plan created.
Case 1: When you are creating a new Test Plan.
When you create a new Test Plan, configure it for reporting to the Test Manager.
In the New Provar Test Plan screen, in the Reporting section, select the Test Manager (TM) check box.
Note: The Connection Name (TM) is the name given to the Test Manager connection by the user.
Case 2: If you already have a Test Plan created.
Double-click on an existing Test Plan. The Test Plan Editor is displayed.
Click “+” (i.e. Add a new feature in Test Plan). And, select the Test Manager(TM) reporting option.
Note: Once this TM reporting is configured, next time when you run or execute the Test Manager, whichever option you have configured for Test Plan that will work, for example, PDF reporting etc.
In the Test Plan Editor, in the Test Manager section, select the Enable Test Manager Reporting check box. When enabled, once a test execution is completed (passed/failed) for the Test Plan and the associated Test Files, the results are reported to the Test Manager.
Note: You can clear the check box if you don’t want the reporting in Test Manager.
Note: If you have enabled the reporting, but you haven’t uploaded the Test Plan then reporting will not be done. So, the user must first upload the Test Plan to the Test Manager.
What is supported right now
As on date, what Provar supports is listed below:
- Test Plan Execution Failure – If the Test Plan execution fails, the Test Manager’s Test Execution object will be updated to reflect the failure.
- Test Plan Folder Execution – On selecting a Test Plan Folder for execution, the Provar Test Execution will identify the Test Plan hierarchy (levels of Test Plan Folders) to the root Provar Test Plan and execute the selected Test Plan Folders.
- Test Plan Instance Execution – On selecting a Test Plan Instance(s) for execution, the Provar Test Execution engine will identify the Test Plan hierarchy (levels of Test Plan Folders) to the root Provar Test Plan and execute the select Test Plan Instances.
Note: Running tests from Test Builder and Debug modes is not supported by Provar.
Running tests from Provar
As of now for running tests from the Run option is only supported. Once you have created a Test Plan, you can execute it. Click on the Test Plan and click Run. The Test Plan is executed.
Note: The Test Plan is executed and there is no reporting in Test Manager at this stage.
Above: Snapshot of Test Plan executed successfully.
Run with ANT build
When exporting Run Under ANT for a Test Plan you can configure reporting to Test Manager, so that test execution results are reported to Test Manager.
When the user selects the option to Run With ANT for a Test Plan, the user is presented with the Export ANT build file dialog. In the Export ANT build file dialog, on the Details tab; in the Features section, the Enable Test Manager (Test) reporting checkbox is selected by default if the Test Manager connection is configured otherwise it isn’t selected.
How results are uploaded to Test Manager
To understand how results are uploaded to Test Manager, first take a look at the object mapping between Provar Studio and Test Manager when executing a test and uploading the test execution results to Test Manager.
To get the reports in Test Manager, you will need to configure the Test Plan. For example, which test cases have passed, this type of information should be there in Test Manager reports.
So, here are some pre-conditions :
- The Test Plan must be uploaded in Test Manager.
- Enable the configuration in the Test Plan for reporting in Test Manager.
Note: If you have only uploaded the Test plan in Test Manager, it will not work. You will have to enable configuration for reporting in Test Manager.
All the test files of the Test Plan must be uploaded as you need to run the test case(s) , and if you have not uploaded the test case then the report of the test case won’t be there in Test Manager. When you execute the test case, a report will be generated in Test Manager.
Just before the test instance in green, we report it. See the screenshot below.
Above: Snapshot of Test Plan execution.
When you run your test, the report generated in Test Manager has two main parts:
- Test Cycle – Test Cycle has the configuration information only. For example:
which was the machine, what was the OS, who executed it, which Browser, which Environment. All this information is in the Test Cycle.
- Test Execution – This section has the actual information like which test case you executed, for how much time you executed, how much time it took for execution, what’s the status (pass or fail).
In Test Manager, the execution will look as shown below.
Click Test Projects > Test Cycles. If a new session starts, it will show all the details. Then, a new record would be created in the Test Cycle.
For example, here we have clicked on AY Plan 1.
Test Cycle Number is auto generated in Test Manager. Provar doesn’t give this number. You can see the information, such as Plan, Environment, Browser, Resolution, OS Name, OS version, which testing tool, testing tool version etc. So, Test Cycle will show just the configuration information but nowhere it mentions which test case is executed and whether that test case has passed or failed; so for that there is a different object which comes with Test Execution.
Let’s take a look at the complete cycle as an example: For example, a user executed 2 test cases, so for these 2 test cases 2 reports are generated i.e. for these 2 test cases, 2 test execution objects are created.
For example, we have opened the first report.
Note: In future, we’ll be covering Test Step execution level i.e. each and every single test step execution will be reported.
The test step results are not uploaded yet and the test step errors are mapped to the TM Test Execution Comments field.
- General information
- Licensing Provar
- Provar trial guide and extensions
- Using Provar
- API testing
- Behavior-driven development
- Creating and importing projects
- Creating test cases
- Custom table mapping
- Debugging tests
- Defining a namespace prefix on a connection
- Defining proxy settings
- Environment management
- Exporting test cases into a PDF
- Exporting test projects
- Managing test steps
- Namespace org testing
- Provar desktop
- Provar Test Builder
- Refresh and Recompile
- Reload Org Cache
- Running tests
- Searching Provar with find usages
- Secrets management and encryption
- Setup and teardown test cases
- Tags and Service Level Agreements (SLAs)
- Test cycles
- Test plans
- Testing browser options
- Tooltip testing
- Using the Test Palette
- Test Palette introduction
- Control test steps
- List compare
- Read test step
- String test steps
- UI Test Steps
- Using custom APIs
- Callable tests
- Data-driven testing
- Page objects
- Block locator strategies
- Introduction to XPaths
- Creating an XPath
- Label locator strategies
- Maintaining page objects
- Mapping non-Salesforce fields
- Page object operations
- Refresh and reselect field locators in Test Builder
- Using Java method annotations for custom objects
- Applications testing
- Database testing
- Document testing
- Email testing
- Mobile testing
- OrchestraCMS Testing
- Guide in Salesforce CPQ Testing in Provar
- Guide in ServiceMax Testing
- Skuid Testing
- Vlocity API Testing
- Webservices testing
- Introduction to test scheduling
- Apache Ant
- Configuration for Sending Emails via the Provar Command Line Interface
- Continuous integration
- AutoRABIT Salesforce DevOps in Provar Test
- Azure DevOps
- Running a Provar CI Task in Azure DevOps Pipelines
- Configuring the Provar secrets password in Microsoft Azure Pipelines
- Parallel Execution in Microsoft Azure Pipelines Using Multiple build.xml Files
- Parallel Execution in Microsoft Azure Pipelines using Targets
- Parallel execution in Microsoft Azure Pipelines using Test Plans
- Bitbucket Pipelines
- GitHub Actions
- Running a Provar CI task in GitHub Actions
- Remote Trigger in GitHub Actions
- Parameterization using Environment Variables in GitHub Actions
- Parallel Execution in GitHub Actions using multiple build.xml files
- Parallel Execution in GitHub Actions using Targets
- Parallel Execution in GitHub Actions using Test Plan
- Parallel Execution in GitHub Actions using Job Matrix
- GitLab CI
- Travis CI
- Execution Environment Security Configuration
- Parallel Execution
- Running Provar on Linux
- Salesforce DX
- Team foundation server
- Version control
- Salesforce testing
- Adding a Salesforce connection
- Assert Page Error Messages on Add/Edit Product
- Dynamic Forms
- Internationalization support
- List and table testing
- Salesforce Release Updates
- Salesforce Lightning Testing
- Salesforce Lightning Web Component (LWC) locator support
- Salesforce console testing
- Visualforce Testing
- Performance Best Practices
- Testing best practices
- Configurations and permissions
- Error messages
- Administrator has blocked access to client
- macOS Big Sur upgrade issue
- Resolving failed to create ChromeDriver error
- Resolving Jenkins license missing error
- Resolving metadata timeout errors
- Test execution fails – Firefox not installed
- Update to Opportunity field validation behaviour
- Licensing, installation and firewalls
- Test Builder and test cases
- Release notes
- Version 2.8.0 Winter ’23
- Version 2.7.2
- Version 2.7.0 Summer ’22
- Version 2.6.3
- Version 2.6.0 Spring ’22
- Version 2.5.0 Winter ’22
- Version 2.4.1
- Version 2.4.0 Summer ’21
- Version 2.3.1
- Version 2.3.0 Spring ’21
- Version 2.2.1
- Version 2.2.0 Winter ’21
- Version 2.1.1
- Version 2.1.0 Summer ’20
- Version 2.0.5
- Version 2.0.4
- Version 2.0.3 Spring ’20
- Version 1.9.12
- Version 1.9.11