One of the big challenges of Salesforce test automation is keeping up with the changes that Salesforce delivers in each major release (Spring, Summer and Winter) and any other changes that get delivered in between. Provar spends a lot of test and engineering time to ensure that we deliver a Provar release where:
- Provar is compatible with every new Salesforce release
- Is available when you start testing with the Salesforce sandbox preview
- Lets you focus on testing new functionality and not updating existing tests to work with the latest Salesforce changes.
Let’s start by identifying two of the key problems that result in many testers struggling with Salesforce releases.
Problem 1: Test tooling/platform mismatch
- The developer doesn’t control how the source is rendered for a Salesforce page. That page renderer is managed and evolved by Salesforce.
Here’s a simplified picture of how Salesforce renders a web page.
Diagram of Salesforce Architecture
Starting from the bottom of the diagram:
- A Salesforce page is built around metadata (page layout definitions, objects and field definitions) and populated with business data (account names, opportunity close dates, mailing addresses, etc.).
- The business data and metadata are processed by a page renderer (Classic, Aura, or LWC) that creates the web page source.
- The web page source is interpreted by a web browser, with the results being displayed on a screen.
Salesforce controls the bottom two layers, defining the form and structure of the page and how the page source code is rendered.
For in-house and deployed enterprise web applications, developers can control how the source code is rendered and can define elements (like uniqueIDs) that make setting up test automation much easier.
Salesforce controls and changes how the page is rendered, so developers are not in control of the things that could make a test more resilient to changes. This is why tests built on rendered source (the presentation layer) are more prone to breaking with Salesforce updates.
We’ve provided a summary some of the key differences between Salesforce, enterprise and in-house web app development that affect testing:
Problem 2: Strict time limits for compatibility testing
The other important piece that the Salesforce developers don’t control is the amount of time they have to deliver compatibility with each new Salesforce update. The sandbox preview is the first chance most customers get to check out the newest release. Previews last from 3 – 4 weeks and then it’s GA. Customers have those weeks to:
- Run their typical regression test to identify potential issues with their customizations (including applications)
- Identify the tests that break
- Determine if it’s a mapping problem (because of a Salesforce change)
- Fix the mapping and rerun the tests
- Identify remaining errors as customization bugs
- Fix the bugs and rerun the tests
The size of the regression test grows along with the amount of customization, making it more and more difficult to fit compatibility testing into the sandbox preview window. Eventually, this builds into what Provar calls the rework spiral. This happens when companies spend more time maintaining test compatibility than adding test automation to ensure the quality of new features and new applications. The cycle repeats with every release and the problem gets worse as the size of the regression test suite increases. The net result is that customers are quickly faced with a choice between increasing testing resources to keep up or accepting reduced coverage and more risk.
The graphic below shows the typical Salesforce release schedule. Partners get an early release 4 to 5 weeks prior to a sandbox preview, which is the first time customers can begin testing the release. GA happens very quickly after the sandbox preview, giving customers a limited time to fix their tests and identify any actual errors.
Pro-Tip #1: Plan Ahead – The rework spiral problem is only apparent after an organization has decided that they can’t get everything done with manual testing and have made the decision to move to automated testing. In fact, dealing with a growing number of breaking tests typically doesn’t get noticed for a few Salesforce release cycles, because it takes time to: a) flag this as something that is going to happen regularly, and b) have enough tests to decide that it’s a problem. At that point, there’s been a significant amount of time, money and effort involved in building out the test automation. Most of the work isn’t transferable to another platform. So plan your automation well and choose your tools carefully.
How Provar delivers Salesforce release compatibility and unbreakable tests
Solution Step 1: Testing based on the Salesforce metadata model
Provar was engineered specifically for Salesforce with a key design criteria of resolving the test tooling/platform mismatch. Instead of using the rendered web page source to build a test, Provar uses the information in the Salesforce metadata model, combined with metadata (including customizations) from the org being tested, to build a test. Here’s a graphic showing how Provar works with Salesforce.
Provar Test Automation Diagram
Building a Salesforce test in Provar involves:
- Connecting to the org being tested and using Salesforce APIs to collect the Salesforce metadata (including any customizations) that define the page layout,object, and field definitions for that org
- Caching the metadata in Provar and organizing the related test step data (unique Salesforce field identifiers, field interactions and data values)
- Executing Provar’s version of a page renderer – the Provar Salesforce layout engine – to automatically create a fully configured test step for any Salesforce page element (as simple as a single mouse click to add a test step)
How is this different from other solutions?
- Page layout definitions, object and field definitions, interactions, Salesforce field identifiers all exist below the web page source level, and change much less frequently. (Salesforce needs these elements to be stable or risk breaking existing applications and customizations.)
- Provar maintains all of the code used to map Salesforce page elements, predict interactions, understand page layouts and automatically generate and execute test steps in Provar. With other web application testing tools, the tester maintains all of the code that defines web page mapping and test step creation.
Solution Step 2: Maintaining Salesforce Release Compatibility
As part of the Salesforce Monorail program, Provar gets access to Salesforce releases 4 – 5 weeks prior to Sandbox previews. During this time, Provar runs approximately 2500 tests to identify where Salesforce has made changes that could potentially affect Provar’s test building applications. The changes can be large (changing from Classic to Lightning) or small (shifting anchors on a Sales console page from Aura to LWC). After evaluating the changes and results of the regression tests, Provar updates its application to handle these changes. Here’s the same diagram as before, with Provar’s compatibility testing and Provar customer testing added.
The net effect: Provar maintains all of the code related to creating a test in Salesforce so that:
- Provar customers don’t need to check tests for release compatibility
- Test failures identified in a sandbox preview are more likely to be real bugs (vs. compatibility bugs)
- Rework spirals can be avoided
- Companies can confidently meet Salesforce GA deadlines
Pro-Tip #2: Analyze other tools for compatibility – Do other test tool vendors get access via Monorail and can they provide compatibility? Yes, other vendors are part of the Monorail program. But, because the test automation code, and most importantly the web page element mapping, is created by the test builder (and individual test builders often create different web locators for the same element), all of that mapping code can only be maintained by the individual customer. So, the vendor has no way of providing release to release test compatibility.
Want to find out how to get out of the rework spiral? Schedule a demo and let’s talk about how to make moving to a new Salesforce releases hassle free.