Partners, customers, and colleagues often ask me how we decide what features and changes we support in the coming Salesforce Releases Schedule. I thought I’d share some inner secrets about how we went about this for the Spring ’21 release and every release for the last two years.
Before we get into the details, let’s first consider the main changes that come from Salesforce:
- Three releases per year, updates to existing products. These changes fall into one of the following sub-categories:
- Changes that are immediately effective/enabled
- Changes that an admin or developer needs to configure/implement
- Changes that are enabled by a future Release Update (see below) or are now being auto-activated with the release
- Release Updates that are time-activated or can be enabled and tested at your leisure
- Acquisition of new technologies by Salesforce
I’m focusing primarily on the first item for this article about the Salesforce release schedule. For the others, we cope with these as part of our regular product release planning.
Now let me share with you when we start testing Salesforce releases, as Michael Caine said…
“Not a lot of people know that.”
– Michael caine
Yes, since 2018, we’ve been invited to test Salesforce releases scheduled weeks before anyone else. We’ve used that time to provide earlier feedback on breaking changes and even helped with reporting bugs before they reached pre-release or sandbox preview. Cool huh?
During this process, we focus first on running our c.5000 regression tests, reporting unusual issues to Salesforce, and, where necessary, amending Provar to cope with the change. With so many tests to run, we prioritize the areas most likely to change, i.e., Lightning UI, first to get results as quickly as possible. Still, we’re continually extending this test pack to include Field Service, CPQ, and Industry Cloud solutions.
It’s worth noting that a minority of changes require the user to make changes to Provar test cases, for example, where a new transition screen has been added and needs specific user interaction. (For example, with the Spring ’21 release, there are UI changes for Lightning Web Components.) As we can’t fix these for you (we deliberately don’t hold your test cases on our servers), we capture these and share them with you in our regular Salesforce release Webinars, Videos, Blogs, and Twitter. I encourage you to make use of these resources!
Going back to the release timeline above, we aim to release an update to the Salesforce Business Scenario Testing (BST) team as early as possible after the pre-release starts. This helps our customers on the BST program get valuable feedback weeks before the sandbox preview on potential breaking changes.
Up to three weeks before the sandbox preview starts, we finally get our hands on the draft release notes – the same as you do (though we check a few days earlier than they are announced). If you’re unaware, useful Trailhead content for Release Planning walks you through what you should consider as a team and how to track changes. We follow something similar already, so I wrote this blog first!
We take these changes and prioritize them as follows:
- Breaking changes that affect or may affect test execution
- Breaking changes that affect or may affect test mapping/creation
- New features that customers are likely to want to create new tests for but are not enabled by default
- Changes and new features that are unlikely to be tested
- Changes and new features that do not impact Provar but may be interesting for future backlog support
From this prioritized list, we will address all the items discovered with Priority 1 before our next release. We aim to ensure you can focus on reporting issues with your application, not ours, when you start sandbox previewing. Where we do not have first-class support for a feature, or you have chosen to use page objects or custom APIs, you sometimes need to amend your custom mappings.
We raise internal change requests for the Priority 2 items. Where we have the capacity two weeks before delivery, and the change is of low regression risk, we will always try to include the change. Any Priority 2 changes we can’t include due to regression risk or complexity get added to our product roadmap for delivery over the next couple of Provar releases and are prioritized alongside our planned feature roadmap. Sometimes, we push other scheduled features back or pull forward changes in a related area that would benefit from delivery simultaneously.
Finally, we will raise change requests for Priority 3 items, but these stay on the backlog until there is a customer or business demand to support the feature. In most cases, Provar customers can still create and execute tests for these Priority 3 features. Still, we aren’t running our daily regression for them, and the tests depend on the user selecting robust locators rather than Provar suggesting the best locator for you.
If you may be interested in the Priority 4 and 5 items, we will cover them in our release webinars, videos, and blogs. For example, the change to rename Critical Updates to Release Updates was important for awareness. Still, it made no difference to most users unless you’re using Provar to automate interaction with the Salesforce Setup as an RPA function.
Finally, I’d like to remind you that Provar can test many Salesforce features intelligently and robustly using metadata and our unique understanding of Salesforce as the only tool dedicated to Salesforce first and foremost. In addition, we also provide API testing, pro-code, and our ProvarXTM page object integration to help you create tests for any other scenario as part of your end-to-end test strategy. Where we don’t have official support for a feature or application, 99% of the time, we still test using ProvarXTM.
Suppose you’ve questioned how we decide which features to include and how little time to implement such changes. In that case, I hope you’ve found this article on how Provar plans for the salesforce release schedule exciting and helpful. If you’d like to comment on changes we make or have specific questions on a feature, feel free to contact us.
