I often get asked by partners, customers and even colleagues how we decide what features and changes we support in Salesforce. I thought I’d share some inner secrets about how we went about this for the Spring ’21 release, and actually every release for the last two years.
Before we get into the detail, let’s first consider the main changes that come from Salesforce:
- Three releases per year, updates to existing products. These changes fall into one of the following sub-categories:
- Changes that are immediately effective/enabled
- Changes that an admin or developer needs to configure/implement
- Changes that are enabled by a future Release Update (see below) or are now being auto-activated with the release
- Release Updates that are time activated or can be enabled and tested at your leisure
- Acquisition of new technologies by Salesforce
For the purposes of this article I’m focusing primarily on the first item, for the others we cope with these as part of our regular product release planning.
Now let me share with you when we start testing Salesforce releases, as Michael Caine said…
“Not a lot of people know that.”– Michael caine
Yes, since 2018 we’ve been invited to test Salesforce releases weeks before anyone else. We’ve used that time to provide earlier feedback on breaking changes and even helped with reporting bugs before they reached pre-release or sandbox preview. Cool huh?
During this process we focus first on running our c.5000 regression tests, reporting unusual issues to Salesforce, and where necessary, amending Provar to cope with the change. With so many tests to run, we prioritise the areas most likely to change, i.e. Lightning UI, first to get results as quickly as possible but we’re continually extending this test pack to include FIeld Service, CPQ and Industry Cloud solutions.
It’s worth noting that a minority of changes require the user to make changes to Provar test cases, for example, where a new transition screen has been added and needs specific user interaction. (For example, with the Spring ’21 release, there are UI changes for Lightning Web Components.) As we can’t fix these for you (we deliberately don’t hold your test cases on our servers), we capture these and share them with you in our regular Salesforce release Webinars, Videos, Blogs and Twitter. I encourage you to make use of these resources!
Going back to the release timeline above, we aim to release an update to the Salesforce Business Scenario Testing (BST) team as early as possible after the pre-release starts. This helps our customers on the BST program get valuable feedback weeks before the sandbox preview on potential breaking changes.
Up to three weeks before the sandbox preview starts, we finally get our hands on the draft release notes – the same as you do (though we check a few days earlier than they are announced). If you’re not aware, there’s useful Trailhead content for Release Planning that walks you through what you should consider as a team and how to track changes. We follow something similar already, which is why I wrote this blog in the first place!
We take these changes and prioritize them as follows:
- Breaking changes that affect or may affect test execution
- Breaking changes that affect or may affect test mapping/creation
- New features that customers are likely to want to create new tests for but are not enabled by default
- Changes and new features that are unlikely to be tested
- Changes and new features that do not impact Provar but may be interesting for future backlog support
From this prioritised list, we will address all the items discovered with Priority 1 before our next release. Our goal is to ensure when you start sandbox preview you can focus on reporting issues with your application and not ours. Where we do not have first-class support for a feature, or you have chosen to use page objects or custom APIs, you will sometimes need to amend your custom mappings.
We raise internal change requests for the Priority 2 items. Where we have capacity in the two weeks before delivery and the change is of low regression risk, we will always try and include the change. Any Priority 2 changes we can’t include due to regression risk or complexity get added to our product roadmap for delivery over the next couple of Provar releases and are prioritized alongside our planned feature roadmap. Sometimes this means we choose to push other scheduled features back or pull forward changes in a related area that would benefit from delivery at the same time.
Finally we will raise change requests for Priority 3 items but these stay on the backlog until there is customer or business demand to support the feature. In the majority of cases Provar customers can still create and execute tests for these Priority 3 features, but we aren’t running our own daily regression for them and the tests are dependent on the user selecting robust locators rather than Provar suggesting the best locator for you.
For the Priority 4 and 5 items, if we think you may be interested in them we will cover these in our release webinars, videos and blogs. For example, the change to rename Critical Updates to Release Updates was important for awareness but makes no difference to most users, unless you’re using Provar to automate interaction with the Salesforce Setup as an RPA function.
Finally, I’d like to remind you that Provar is able to test many Salesforce features intelligently and robustly using metadata and our unique understanding of Salesforce as the only tool dedicated to Salesforce first and foremost. In addition we also provide API testing, pro-code and our ProvarXTM page object integration to help you create tests for any other scenario as part of your end-to-end test strategy. Where we don’t have official support for a feature or application, 99% you can of the time still test using ProvarXTM instead.
If you’ve ever questioned how we decide which features to include and how little time there is to implement such changes, then I hope you’ve found this interesting and helpful. If you’d like to comment on changes we make or have specific questions on a feature, feel free to get in touch.