We're here to help


This includes information regarding a solution to setting up integration from Flosum to execute a Jenkins CI Server Job to execute a suite of Provar Tests and write the test results back to a custom object in Salesforce. The process is broadly the same for other applications, such as Agile Accelerator or if you want to call Provar tests from within a custom application inside Salesforce.

A video demonstration and overview of configuring our Provar Test Results package is also available from the following link: http://bit.ly/ProvarFlosumTI


Suggested architecture

In order to configure your DevOps process with Flosum the following three main steps need to be completed which are elaborated further in this document:

  1. Setup and secure a publicly accessible server instance to host the CI server, such as  Jenkins. We’ve used AWS for this in the documentation below. If you use your own server, make sure it’s within your network DMZ and your firewall is permitted traffic through on port 8080 (or any other port you choose to use for the CI Server).
  2. Install the Provar Test Integration package into the Flosum Org where you wish to save test results and configure a new process to invoke the Provar CI job.
  3. Configure your Provar Test Cases with Finally Steps to save test results back to your Flosum org using the Provar Test Execution custom objects.


AWS configuration

If you’re using a locally hosted Jenkins instance please ensure it is within your corporate DMZ and can accept incoming connections from Salesforce; you can skip to the next section. The range of Salesforce IP addresses is long and ever changing so we recommend a cloud hosted instance.

An AWS instance should be configured by following the AWS Set Up a Jenkins Server guide. Note the full public AWS DNS JENKINS_URL must be used not just the IP address:

e.g. http://ec2-user@ec2-34-245-70-169.eu-west-1.compute.amazonaws.com:8080

After completing your setup ensure you can access your new Jenkins Admin screen remotely using the JENKINS_URL from your local desktop browser before continuing and not just on the AWS instance using localhost:8080. If this fails you need to check your AWS Configure Security Group and ensure it’s been applied to your AWS instance. Do not proceed until this is working.

If installing onto a Windows Server you will also need to also create an Inbound Port Forwarding rule on Windows Firewall for port 8080. Do not restrict source IP access unless you plan to whitelist every Salesforce IP address.

It is your responsibility to lock down this AWS instance and Jenkins to meet your corporate security standards. The instance must be accessible from Salesforce.


Jenkins configuration

Follow the Provar Setting Up Continuous Integration guide.

After configuring the server the CSRF Protection needs to be disabled in the Manage Jenkins -> Configure Global Security. The default settings for the Access Control should be left as below until you have your integration working, and then can be locked down using Matrix Based Security.

Note: By enabling Read Only Anonymous access you can allow non-authenticated users to inspect the results of the build action. Disable this if you do not want to allow this to be publicly visible to anyone with the Jenkins server URL, and set up any additional non-admin user access you require instead.

We recommend not using your Jenkins Admin user credentials for triggering remote builds, and creating a new user specifically for this purpose, using Manage Jenkins -> Manage Users to add a new user.

For the Jenkins user you want to use to trigger tests remotely, make a note of the Username and API Token to be used. The password is notrequired for API access.

Note: You need to login as the user to be used and click the Show API token BEFORE you restrict access if using the Matrix based security.

While you can integrate with the Jenkins Admin user but we strongly recommend you create a new user identity in the Manage Jenkins -> Manage Users and limit their execution to execute build jobs only once you have your integration working and have captured the API Token as above for the new user:

Deploy your test cases to your Jenkins CI Server, or integrate with your Version Control repository within the Build job you wish to trigger.


Provar deployment

Depending on your AWS Instance you will either need to follow the standard install for either Windows or Linux to setup and license Provar for your execution environment.

You should also read carefully and follow all relevant steps in the Provar DevOps guide.

When configuring your Provar Build Job ensure the job is made available as a Remote Build and make a note of your Build Authentication Token and the Build URL. For remote invocation we recommend creating an additional user for this purpose, see Jenkins Configuration section. For linking Test Suite records back to Flosum use the buildWithParameters version of the Build URL.

If you wish Provar to write back your Flosum Metadata Log you also need to enable parameter passing for this job with the parameter name of EXTERNAL_ID.

You can also add the following optional parameters if you wish to override the Test Case Path and/or the ANT target to be executed.

Salesforce configuration

Within Salesforce there are the following changes required:

  • Install Provar Test Integration Package, consisting of:
    • A Test Suite Execution custom object which summarises the execution results for a particular deployment.
    • A Test Case Execution custom object to which individual test cases update with the Provar test result for that test case. The Test Cases are related to a parent Test Suite Execution record.
    • A hierarchical Custom Setting Provar CI Settings to parameterise the Jenkins connection and build job details/ values.
    • An Apex Class CIJobNotificaton which includes both an invocable apex method invokeCIJob() and makes an asynchronous call using values specified in the Custom Setting above using httpCallout(). Both these methods are global. InvokeCIJob() can be called from Process Builder or CIJobNotification.httpCallout() can be called from an Apex class or Trigger.
    • Some helpful reports and dashboards to get you started on visualising your results.
  • Create a locally implemented Process Builder Process that calls Provar_te__CIJobNotification.invokeCIJob, passing an External Id parameter for callback purposes to link test results. For example, to link it to the Flosum Metadata Log object, when a deployment has been successfully completed.
  • Create a Remote Site Setting for the CI Server Endpoint URL value, e.g. http://ec2-34-242-64-169.eu-west-1.compute.amazonaws.com:8080
  • Finally, if you want to link our test results to the Flosum Metadata Log record id you need to create a new Lookup relationship from Test Execution Suite object to the Flosum Metadata Log object. When this is populated in the Provar test case the test results will automatically become visible from Flosum.


Package implementation

  1. Install the Provar Test Integration package to the org where you want to trigger builds and write back test results. When prompted ensure you grant permission to the user profiles used in your Provar Test Cases so they have read/write access, plus any admin users you use in Flosum. The Provar Test Integration Results is available from the AppExchange: https://bit.ly/ProvarTIR. IMPORTANT! When installing the TestResults package ensure that the user profile that will be used in the Provar Connection has full access to the package.
  2. Create a new Process Builder Process on Flosum Metadata Log as follows and set the Object condition as per the screenshot below. Alternatively, you can call the global Provar_te__CIJobNotification.httpCallout(String externalId) method directly from an Apex trigger if you prefer to implement a coded solution and skip to step 5.

  1. Setup the criteria for a completed and successful Flosum deployment. In this case we’ve chosen when deployments are completed and successful.

  1. Add an Apex action to call the invokeCIJob method, passing the Metadata_Log ID as the buildParams parameter.

You can pass the following optional parameters if you wish, using a comma to separate each parameter in the buildParams result. Use a Formula field to construct the result:

  • Test Case Path – A sub-folder for the group of tests to be executed. You’ll need to ensure your ANT build.xml script is updated to utilise the parameter.
  • ANT Target – An alternate ANT target to be executed. This can be used to configure different browsers, screen sizes or more complex selection of tests to be executed. If not specified the default target will be executed.

  1. Add a Remote Site Setting for the CI Server Endpoint URL value to allow Salesforce to make callouts to the CI server, e.g. http://ec2-34-241-67-179.eu-west-1.compute.amazonaws.com:8080
  1. Before we can call our Jenkins build we now need to pass the credentials are URL to be used. Within Salesforce Setup menu, select Custom Settings and click the Manage link next to the Provar CI Settings entry:

Now click the New button at the top of the page to set the Default Org level settings and enter the following information you captured when setting up Jenkins and the Provar Build job:

  • CI Server URL: URL for accessing the Jenkins build job you set up earlier, enter the Build URL
  • CI Username: The Username authorised to execute the Jenkins job, which should be restricted to only perform this action.
  • API Token: The corresponding API Token noted earlier for the CI Username
  • Authentication Token: The Authentication Token from the Jenkins Job to be executed.

  1. Finally, if you want to link the test results to the Flosum Metadata Log record id you need to create a new Lookup relationship from Test Execution Suite object to the Flosum Metadata Log object. When this is populated in the Provar test case the test results will automatically become visible from Flosum.

Provar integration with Salesforce/Flosum

A sample project can be found at https://github.com/rclark-provar/ProvarTestResults which writes results back to the Test Integration package in a demo sandbox.

For your own purposes you will need to modify your test cases as follows using Provar, or amend those provided in the project above to point to your own Salesforce instance:

  1. User is required to create a Setup Test Case. This test case will insert a Test Suite Execution record in Salesforce and make it available for the Test Run scope:

Within the Create Object step ensure you set the following values:

  • Metadata Log Id : With the External Id passed into Jenkins and serialised by the ANT job.
  • Test Suite Name : Any relevant name which identifies the Test Suite, such as the date and time plus some description.

After the Test Suite Execution record is created, it should return:

  • TestSuiteExecutionId : Salesforce object id to identify record
  • Set Results Scope to Test Run to make this visible to all test cases
  1. Create a new Test Case which is going to insert entries to the Test Case Execution record in Salesforce. Make this test callable. Within the Create Object steps ensure you set the following values:
    • Test Suite Execution: Will provide TestSuiteExecutionId created in previous step
    • Name: Current Test Case name can be retrieved {TestCaseName()}
    • Status: Pass/Fail
    • StartDateTime: Optional. You may want to capture this at the very start of your test in a Variable step and pass it as a variable here.
    • EndDateTime: Optional. Use the Expression {NOW}
    • Comments: Optional. Any additional information you want to record in the Test Case Execution record.

  1. Modify your existing tests to add a Finally step, drag and drop your callable test you created above into the Finally block and set the following input parameters:
    • Status: We recommend passing either the {TestCaseSuccessful()} or {TestCaseOutcome()} expressions. The first will return true/false the latter will return Successful/Skipped/Failure. In our callable test above we’re use TestCaseSuccessful.
    • TestCaseName. The name of the test case which can be extracted using the expression {TestCaseName()}

  1. After completing your configuration and saving either to the build server or Version Control repository click the Build Job button in Jenkins. You need to set a valid Flosum Metadata Log Id for the EXTERNAL_ID to link it to the Flosum job that triggered the build. You can leave this empty if you do not wish to link the Test Suite to a Flosum job.

In the case of Jenkins the execution is triggered from Flosum:

  1. A variable, representing the Metadata_Log_Id should be passed to Jenkins as the External ID in the steps earlier. This variable will then be sent to the ANT build.xml file as a parameter.
  2. Create a Provar build.xml using the steps documented in Generating a Build File. Edit the build.xml file, to use an echo to write the External Id in a file in the templates folder under the project so that Provar can import the value and link Test Suite records back.

  1. If you are using the optional parameters in Process Builder to pass the test case Path and/or ANT target make the following additional changes to your build.xml:
    • For test case Path, replace the with
    • For alternate ANT targets, copy the entire … block and paste underneath, updating the target name to be unique and change the test execution parameters as desired.
  2. Using the Read test step of Provar, we will add a step to read the content from the file we wrote to in the first step while inserting Test Suite record.

Testing checklist

We recommend you test each part of the process before running end to end. Ensure each of the following works to diagnose any issues you have:

  1. The server where Jenkins is deployed can be reached remotely, or at least the Jenkins instance accessed. If this is failing your AWS Security Rules or Server Firewall may be blocking connections. Ensure the Jenkins port (e.g. 8080) is permitted for Inbound Connections and re-test.
  2. The Provar job on the Jenkins server can be executed within the Jenkins admin console. If this is failing examine the errors, fix the root causes and re-test.
  3. The Provar job on the Jenkins server can be executed remotely from the command line, using curl. If this is failing examine the error returned. The most likely cause is the Jenkins Global Security. Try opening up access and re-testing to diagnose the issue.
  4. A Flosum Metadata Log entry triggers a new Jenkins build for the Provar job. You can mimic the creation of a Flosum Metadata Log by executing the following code in the Developer Console using the Execute Anonymous Apex option.

If this is not showing any errors check the Provar CI Settings have been created, and that the value populated are correct. If you’re still having issues, contact Salesforce Support and request that Debug Logging is enabled for the Provar Results package and re-test.

Hopefully, you now have a fully working end-to-end process to execute your Flosum based Continuous Delivery Cycle, well done! However if the last step above is failing ensure you have enabled Debug Logs in Salesforce, or are using the Developer Logs and check for error messages.