2 Keys to Simplifying Salesforce Sandbox Preview Testing

Published by Alexander Sherwood, January 27, 2022
Salesforce, Tips
2 Keys to Simplifying Salesforce Sandbox Preview Testing

One of the big challenges of Salesforce test sandbox automation is keeping up with the changes that Salesforce delivers in each major release (Spring, Summer, and Winter) and any other changes that get delivered in between. Provar spends a lot of testing and engineering time to ensure that we deliver a Provar release where:

  • Provar is compatible with every new Salesforce release 
  • Is available when you start testing with the Salesforce sandbox preview
  • Lets you focus on testing new functionality and not updating existing tests to work with the latest Salesforce changes.

Identifying Two Key Problems With Salesforce Releases.

Problem 1: Test Tooling/Platform Mismatch

  • The developer doesn’t control how the source is rendered for a Salesforce page. That page renderer is managed and evolved by Salesforce.
  • Salesforce regularly makes changes to how a page is rendered, which directly affects the HTML, CSS, Javascript, and DOM that most test automation tools rely on to build tests.

Here’s the problem: most Salesforce test automation tools use the same approach to testing Salesforce that they use for testing any web application. Specifically, those tools map page elements and build tests based on information about the rendered page – the HTML, CSS, javascript, and DOM of the page. While this works exceptionally well for most web applications, it’s a problem for Salesforce testing for two key reasons:

Here’s a simplified picture of how Salesforce renders a web page.

Here’s a simplified picture of how Salesforce renders a web page to test sandbox
Diagram of Salesforce Architecture

Diagram of Salesforce Architecture

Starting from the bottom of the diagram: 

  • A Salesforce page is built around metadata (page layout definitions, objects, and field definitions) and populated with business data (account names, opportunity close dates, mailing addresses, etc.). 
  • The business data and metadata are processed by a page renderer (Classic, Aura, or LWC) that creates the web page source.
  • The web page source is interpreted by a web browser, with the results being displayed on a screen.

Salesforce controls the bottom two layers, defining the form and structure of the page and how the page source code is rendered. 

For in-house and deployed enterprise web applications, developers can control how the source code is rendered and can define elements (like uniqueIDs) that make setting up test automation much easier. 

Salesforce controls and changes how the page is rendered, so developers are not in control of the things that could make a test more resilient to changes. This is why tests built on the rendered source (the presentation layer) are more prone to breaking with Salesforce updates. 

We’ve provided a summary of some of the key differences between Salesforce, enterprise, and in-house web app development that affect testing:

 a summary of some of the key differences between Salesforce, enterprise, and in-house web app development that affect sandbox testing

Problem 2: Strict Time Limits for Compatibility Testing

The other important piece that the Salesforce developers don’t control is the amount of time they have to deliver compatibility with each new Salesforce update. The sandboxed preview is the first chance most customers get to check out the newest release. Previews last from 3 – 4 weeks and then it’s GA. Customers have those weeks to:

  • Run their typical regression test to identify potential issues with their customizations (including applications)
  • Identify the tests that break
  • Determine if it’s a mapping problem (because of a Salesforce change)
  • Fix the mapping and rerun the tests
  • Identify remaining errors as customization bugs
  • Fix the bugs and rerun the tests

The size of the regression test grows along with the amount of customization, making it more and more difficult to fit compatibility testing into the sandbox preview window. Eventually, this builds into what Provar calls the rework spiral. This happens when companies spend more time maintaining test compatibility than adding test automation to ensure the quality of new features and new applications. The cycle repeats with every release and the problem gets worse as the size of the regression test suite increases. The net result is that customers are quickly faced with a choice between increasing testing resources to keep up or accepting reduced coverage and more risk. 

The graphic below shows the typical Salesforce release schedule. Partners get an early release 4 to 5 weeks before a sandbox preview, which is the first time customers can begin testing the release. GA happens very quickly after the sandbox preview, giving customers a limited time to fix their tests and identify any actual errors. 

Pro-Tip #1: Plan Ahead – The rework spiral problem is only apparent after an organization has decided that they can’t get everything done with manual testing and have made the decision to move to automated testing. Dealing with a growing number of breaking tests typically doesn’t get noticed for a few Salesforce release cycles, because it takes time to: a) flag this as something that is going to happen regularly, and b) have enough tests to decide that it’s a problem. At that point, there’s been a significant amount of time, money, and effort involved in building out the test automation. Most of the work isn’t transferable to another platform. So plan your automation well and choose your tools carefully.

How Provar Delivers Salesforce Release Compatibility and Unbreakable Tests

Solution Step 1: Testing based on the Salesforce metadata model

Provar was engineered specifically for Salesforce with key design criteria of resolving the test tooling/platform mismatch. Instead of using the rendered web page source to build a test, Provar uses the information in the Salesforce metadata model, combined with metadata (including customizations) from the org being tested, to build a test. Here’s a graphic showing how Provar works with Salesforce.

Provar Test Automation Diagram in Salesforce Test Sandbox

Provar Test Automation Diagram

Building a Salesforce test in Provar involves:

  • Connecting to the org being tested and using Salesforce APIs to collect the Salesforce metadata (including any customizations) that define the page layout, object, and field definitions for that org
  • Caching the metadata in Provar and organizing the related test step data (unique Salesforce field identifiers, field interactions, and data values)
  • Executing Provar’s version of a page renderer – the Provar Salesforce layout engine – to automatically create a fully configured test step for any Salesforce page element (as simple as a single mouse click to add a test step)

How is this different from other solutions?

  • Building a test in Provar does not rely on the rendered HTML, CSS, or Javascript
  • Page layout definitions, object and field definitions, interactions, and Salesforce field identifiers all exist below the web page source level and change much less frequently. (Salesforce needs these elements to be stable or risk breaking existing applications and customizations.)
  • Provar maintains all of the code used to map Salesforce page elements, predict interactions, understand page layouts and automatically generate and execute test steps in Provar. With other web application testing tools, the tester maintains all of the code that defines web page mapping and test step creation.

Solution Step 2: Maintaining Salesforce Release Compatibility

As part of the Salesforce Monorail program, Provar gets access to Salesforce releases 4 – 5 weeks prior to Sandbox previews. During this time, Provar runs approximately 2500 Salesforce test sandbox to identify where Salesforce has made changes that could potentially affect Provar’s test-building applications. The changes can be large (changing from Classic to Lightning) or small (shifting anchors on a Sales console page from Aura to LWC). After evaluating the changes and results of the regression tests, Provar updates its application to handle these changes. Here’s the same diagram as before, with Provar’s compatibility testing and Provar customer testing added.

Here’s the same diagram as before, with Provar’s compatibility testing and Provar customer testing added in Salesforce test sandbox

The net effect: Provar maintains all of the code related to creating a test in Salesforce so that:

  • Provar customers don’t need to check the salesforce test sandbox for release compatibility
  • Test failures identified in a sandbox preview are more likely to be real bugs (vs. compatibility bugs)
  • Rework spirals can be avoided
  • Companies can confidently meet Salesforce GA deadlines

Pro-Tip #2: Analyze other tools for compatibility – Do other Salesforce test sandbox tool vendors get access via Monorail and can they provide compatibility? Yes, other vendors are part of the Monorail program. But, because the test automation code, and most importantly the web page element mapping, is created by the test builder (and individual test builders often create different web locators for the same element), all of that mapping code can only be maintained by the individual customer. So, the vendor has no way of providing release-to-release test compatibility.

Want to find out how to get out of the rework spiral? Schedule a demo and let’s talk about how to make moving to a new Salesforce release hassle-free. 

We use cookies to better understand how our website is used so we can tailor content for you. For more information about the different cookies we use please take a look at our Privacy Policy.

Scroll to Top