How Alm. Brand Uses Provar With GitLab and Docker

“We have never seen a UI testing tool that has been able to assist so much and enable us to make a fully functional test suite within four months from the first time we got access.”

Nicolai Rønnow Iversen
Technical SalesForce Architect, Alm. Brand

How Alm. Brand Leverages Provar at a Glance

150 Provar test cases running nightly within two test orgs. Currently testing Salesforce Sales Cloud and Service Cloud including Console, Lightning testing, Flows, Visualforce, and many integrations.

The Challenge of Using Provar with GitLab and Docker

To support a fully-functional agile software development methodology, the Alm. Brand needed to leverage test automation capabilities with GitLab and Docker to deliver a new release on a 14-day cycle and keep improving multiple times a day.

As part of Alm. Brand’s dedication to delivering the best experience possible for users, quite an extensive automation suite was built up, acting as a regression test suite when promoting code from one test environment to another. When Salesforce made a major update to the user interface, transitioning from classic to Lightning, this suite was no longer compatible. The Alm. Brand team was forced to resort to using manual testing processes which required 120+ hours of manual testing per regression cycle.

The Solution to Manual Testing

After conducting a thorough evaluation of other automated software testing applications, the Alm. Brand worked with House of Test, the self-confessed Rebels of IT, which provide expert test consultancy for clients across Sweden, Denmark, Switzerland, and the U.K. to select and implement Provar to simplify and accelerate the process of software innovation.

Test Case Implementation

Automated tests that rely on UI are naturally slower to execute than tests that rely only on APIs. As Provar supports both UI access and makes SOQL/API integration easy, all tests in Alm. Brand’s automation suite uses a minimum of UI components.

The tests will typically set up the test data using API access, then run a short user scenario with UI components, and finally, verify the results with API access to support the strategy based on the testing triangle. Alm. Brand is able to use the test suite across environments and use the same test with multiple users to verify the difference.

Continuous Integration and Containerization

Even with these optimizations, the regression suite takes a considerable time to run. To cut down on this time, the Alm. Brand in collaboration with the House of Test configured the GitLab pipeline to run several tests in parallel using multiple Docker containers. The parallelization is just limited by how many Docker containers GitLab can handle in parallel.

Alm. Brand designed the test cases to be self contained to provide the option to run all test cases in parallel. Given this, the overhead to set up the test basis for each test case and remove it is won multiple times by being able to execute it in parallel.

GitLab Pipeline

Using Provar’s support for automating tests with Ant, Alm. Brand built a pipeline in GitLab that enables the Alm. Brand teams to work on the automation suite collaboratively and get fast feedback, both when committing new code and when promoting releases to new environments.

The pipeline is based on Docker continuous integration containers that contain the Provar tool, Ant, and Chrome. GitLab uses these containers to run small smoke tests when new code is checked in, followed by a larger suite when code is merged. Every night, a full regression suite is run, in several environments, so every morning, the teams at Alm. Brand can get feedback if any new deployment has caused any regressions.

For versioning of test cases, the Alm. Brand used Gitlab to design a branching structure to create multiple versions of the same test case so the test case in the environments followed the actual code it was testing.

The Results with GitLab and Docker

As a result, Alm. Brand now has a much more robust regression test suite that can give fast feedback, easily be maintained, and is executed automatically. This not only saves Alm. Brand many hours of manual testing for each release, but also increases the confidence of the stakeholders and development teams in Alm. Brand each time new code is released, increasing the quality and the speed of the deliveries.

Stats at a Glance

The Alm. Brand was able to get up and running with Provar in less than four months
By implementing Provar’s automated testing capabilities, the team is able to save 120+ hours of manual testing per regression cycle

“One of the reasons Alm. Brand has been so successful in their implementation that they focused on creating tests to keep maintenance at a minimum.“

“They created tests in an optimal way to reduce future rework, adopting features such as callable tests, API testing, and data-driven testing into their overall architecture to underpin their UI testing. This enabled a fast and efficient delivery of a test suite that will remain effective into the future.”

Piyush Srivastav
Customer Success Manager, Provar

“With Provar, we did not just succeed and create a foundation for our automated testing, but also a foundation for our goals to the CI implementation to support our Agile strategy.

We could support the idea of moving as much as possible of the test down in the test pyramid while keeping our test coverage intact. One of the most essential things that have made this possible has been the way Provar treats its customers, not only helping to solve the problem but also listening and receiving advice from the customer on what can make Provar a better tool.

They are not afraid of being challenged and at the same time bringing their ideas to the table, that part has made communication between Alm. Brand and Provar to one of the leading pillars that has made us reach where we are now in just four months.”

Nicki Bang-Madsen
Technical Tester, Alm. Brand