We were able to increase 12x productivity and one QE wrote the scripts in one week [with Provar].Mohit Kumar, Senior Member, Technical Staff at Salesforce
Last week, Provar hit the Dreamforce stage with members of Salesforce and Alm. Brand Group to discuss “Transforming Test Automation and Quality,” an excellent opportunity for professionals from across the globe to listen to applicable solutions and hear about experiences from two of our customers. For those who were unable to attend, here is a recap.
The session focused on how to measure quality in their software lifecycles, automate tests declaratively, manage test execution, and adopt a QA maturity hub to integrate all measures. Thought leaders included Ben Kim (Senior Manager, Software Engineering) and Mohit Kumar (Senior Member, Technical Staff) at Salesforce, and Nicki Bang-Madsen (Technical Tester) at Alm. Brand Group. Alexa Cash (Director of Customer Success) and Michael Dailey (Director of Solution Engineering) at Provar facilitated the discussion.
Provar opened with an overview of its two products – Provar Manager, our quality hub for managing test cases, test plans, and test results, and Provar Automation, our test automation solution built from the ground up specifically for Salesforce. Both Salesforce and Alm. Brand Group shared customer success stories using Provar’s solutions.
Salesforce: Challenges from the Employee Success Domain Team
Like many mature QA teams, frustrations mounted with Selenium’s lack of reliability and simultaneous increase in QA demand. Provar provided the solution; Ben and Mohit and the team at Salesforce saw increased testing velocity and more reliable releases within 6 months in their V2MOM application regression testing.
The Salesforce team provided a detailed Q&A including an introduction of their Business Technology Team within their Employee Success Domain Team, which supports application testing for internal Human Resources applications, which encompasses 80,000+ users. Their high-demand QA needs included challenges such as a demand for an increased volume of feature releases, lean testing team vs. QA demands, non-scalable manual testing, the failure of Selenium scripts in release regression testing, and problematic Selenium maintenance requirements in a Salesforce release cycle.
After doing some research, we found out about the power of Provar. One of the biggest pain points we had when it came to automation, particularly on Selenium, was maintenance. We were spending 30-40% of our time continually redoing our automation test scripts in Selenium. With Provar, we were able to drastically reduce that entire maintenance effort, so it’s been really a big win for us.Ben Kim, Senior Manager, Software Engineering at Salesforce
The impacts they saw as end users since incorporating Provar Automation as a solution were profound. Here are the results within six months, including onboarding and learning the tool and automation efforts:
- Lower average hours/sprint per person: 88% reduction in test maintenance time per FTE per QA sprint (2 dedicated QEs, 80 hours/person, 2 sprints of maintenance efforts to update and maintain Selenium test cases)
- Faster time to automation: 12x faster; it takes 12 weeks to complete the same number of scripts in Selenium that it takes to automate with Provar (an important factor on this is that the 12 weeks of work work only included UI testing, and the 1 week with Provar included UI and API testing)
Our scripts were failing, and all our effort building them was wasted. So we introduced Provar, we onboarded, and we were incredibly amazed. We couldn’t believe the reduced time we now spent on scripts. We went from spending 3 months to doing the same effort in one week’s time with one human using Provar.Mohit Kumar, Senior Member, Technical Staff at Salesforce
Alm. Brand Group: Increasing Confidence Across the Organization and Beyond
As a longtime Provar customer and user at multiple companies, Nicki Bang-Madsen has seen extensive growth, outcomes and benefits with Provar products, ranging from adoption to interoperability. His team’s results range from increased regression speed, to more nuanced success such as more confident developers.
Alm. Brand Group is Denmark’s second largest insurance group with 700k+ customers and the first financial institution in Denmark to use Salesforce. After incorporating Provar’s solutions, long-term outcomes included:
- Ease of adoption to Salesforce teams: Teams appreciated the similar mindset between Provar solutions and their existing knowledge from working with Salesforce
- More than just regression testing: 90% less time spent on regression testing, and 60% less errors reported by production (outcome was better development ROI and more fun and satisfaction across the teams)
- Testing the gaps: Testing the teams’ end-to-end processes helps catch any gaps and increases trust and satisfaction from both users and stakeholders
- 4x reduction in hot fixes in production: Due to unique Provar capabilities that aren’t accessible by using standard Selenium tools
- Increased reliability that instills confidence and satisfaction: Reliability of tests and more team collaboration provides confidence and allows for faster delivery
[With Provar], by reducing errors, we increased confidence across our teams, stakeholders, end users, and through the entire organization.Nicki Bang-Madsen, Technical Tester at Alm. Brand Group
A big thanks to Salesforce and Alm. Brand Group for partnering with us on this lively discussion. We are grateful for our partners in testing! For more Dreamforce news, connect with Provar on social media.