Architecting Success at Splunk and Alm. Brand

December 04, 2019

At Dreamforce 2019 Provar hosted a fascinating session with Mala Punyani, Test Architect at Splunk, and Nicolai Rønnow Iversen, IT Architect at Alm. Brand. In that session, those architects described their strategies for implementing Provar in their organizations and the benefits they realized.

Note: Salesforce recorded this session, so we’ll update this blog when the video is up.

Although the two architects described very different approaches within their organizations, both achieved incredible reductions in testing effort in a few short months from Provar adoption.

What benefits did Splunk realize after adopting Provar?

Splunk achieved an overall 75% reduction in test effort after 3 months:

Credit: Mala Punyani, Test Architect at Splunk 

Let’s dig into what made Splunk so successful with Provar.

End-to-End Testing (Splunk)

In the session Splunk described how they automated their entire purchase order flow using Provar, including quoting with Salesforce CPQ, order generation in Netsuite, and completion of the Salesforce sales cycle including licensing, service contracts and entitlements.

Below shows the flow automated using Provar:

Credit: Mala Punyani, Test Architect at Splunk

Strategic Test Features

Splunk highlighted the following key features of Provar as particularly valuable to their test strategy.

  • Test data management:
    • Data driven facility with Read/Write via/into Excel
    • APIs to quickly create test data in opportunities, accounts and other objects
    • Usage of SOQL queries to return values and assertions
    • Bulk Update feature via API
    • Control APIs, e.g. Loop for repetitive actions
    • Table creation in Provar for dynamic value assertions
    • Batch execution in Salesforce with the help of Execute Apex API
  • Provar Managed Dashboard
  • End to end:
    • Automated scenarios where multiple systems are integrated and the data flows from one system to other
    • PDF validation
    • Verification of email notification triggered through Salesforce CPQ (Steelbrick) quoting system
    • Service Contract Batch Generation

For more information refer to the Provar product page.

What benefits did Alm. Brand realize after adopting Provar?

Alm. Brand achieved a 90% reduction in hours spent release testing after 6 months. Interestingly they also noted a significant reduction in user-reported errors and the number of hotfixes in production was halved compared to when they had been using Selenium.

Credit: Nicolai Rønnow Iversen, IT Architect at Alm. Brand

Solving Selenium Challenges (Alm. Brand)

Provar is designed to address some of the challenges of testing the Salesforce platform and solve issues where other solutions may struggle, such as reliability and reusability.

Nicolai Rønnow Iversen, IT Architect at Alm. Brand, said:

“We came from a Selenium setup that was extremely time consuming to develop and maintain. After we went from Classic to Lightning in our Salesforce org, we lost our entire test setup. Thousands of hours lost, and we had no way of reusing any of it.”

Alm. Brand described the following benefits of moving to Provar after using Selenium:

  • Faster development time of test cases
  • 3-12 faster test execution than Selenium
  • More resilient to changes in UI

Credit: Nicolai Rønnow Iversen, IT Architect at Alm. Brand

Alm. Brand also noted that, now then they create test cases, it is only the functionality that they want to test that is done in the UI. All the rest is done on on API or SOQL, which allows them to develop test cases much faster.

Nicolai noted,

“The Provar test cases execute 3-12 times faster compared to Selenium, and as an important bonus our tests are much more resilient to changes in the UI.”

For more information on Alm Brand’s approach to test case implementation, take a look at the Case Study: How Alm. Brand Uses Provar With GitLab and Docker to Support a Complex Salesforce Solution.

Parallel Testing with Gitlab and Docker (Alm. Brand)

Another impressive element of Alm. Brand’s test strategy is their use of GitLab pipelines to run several tests in parallel using multiple Docker containers.

Credit: Nicolai Rønnow Iversen, IT Architect at Alm. Brand

Alm. Brand noted that running Provar in Docker containers on GitLab pipelines enabled them to quite easily spin up 9 instances to execute all their main test areas simultaneously. Alm. Brand currently have 150 test cases that execute in less than 2 hours, so they run them all each night to ensure short feedback loops to their developers and configurators.

Nicolai said,

“We develop all our test cases so they are user and Org agnostic, in this way we supply our test the Org and Salesforce user as a parameter.

“Our test cases are in GitLab, and we move them up through the branches in the same pace as we move Salesforce changes, in this way we can test the different versions of Salesforce functionality as we move it up through our Orgs.”

For more information on Alm Brand’s use of Continuous Integration and containerization, take a look at the Case Study: How Alm. Brand Uses Provar With GitLab and Docker to Support a Complex Salesforce Solution.

Provar and Continuous Integration

Jon Robinson of Provar noted,

“It’s great to see the success Alm. Brand has had with GitLab and Docker. Provar aims to support every aspect of a customer’s test strategy including their different release management providers and tools, CI solutions and services.

Thank you, Splunk and Alm. Brand!

All in all a fascinating talk from two companies having success with Provar with very different testing strategies. Thank you to Mala Punyani, Test Architect at Splunk, and Nicolai Rønnow Iversen, IT Architect at Alm. Brand, for their insightful commentary.

Interested to learn more?

If you’d like to know how Provar could help your testing, why not join our upcoming webinar, Provar 101: Introduction to Test Automation? Join us for this introductory webinar on Thursday 12th December as we demonstrate how to achieve test automation that’s easy to set up, simple to maintain and readily integrated into your DevOps pipeline. Click here to register.