Setting up continuous integration
Once the Build File is ready and Version Control has been set up (if wanted), tests can be scheduled using a Continuous Integration Server such as Jenkins.
The role of the CI Server is to run the regression pack (from a local machine or via a version control repository). The CI Server will trigger the Provar automation pack on a defined schedule or based on an external trigger, such as a new build or deployment.
Start by downloading and installing Jenkins.
To download Jenkins, visit the Jenkins website. Click the Download button and select the file which suits your operating system.
To install Jenkins, first move the downloaded Jenkins.war file to an appropriate directory, e.g. C:\Program Files\Jenkins:
Then open command prompt in this directory by clicking on the filepath and typing cmd:
Once command prompt has opened, type and run the command java –jar Jenkins.war.
Once the command has run, a password will be generated for logging into Jenkins:
In this example the password is cd3f375e0baa47f4b4a5591f1e920bfe.
Note that this password can also be checked by referring to the InitialAdminPassword.txt file which has been created in the \.jenkins\secrets folder:
Next, open your browser and navigate to URL: localhost:8080:
The following screen will appear:
Enter your password then click the Continue button.
On the next screen, choose the option Select plugins to install:
Some plugins will be defaulted already. These can be left as defaulted.
Make sure that the following plugins are ticked:
Ant Plugin (under Build Tools):
JUnit Plugin (under Build Analysis and Reporting):
If you want Jenkins to download the regression pack from your Version Control repository, also tick the relevant plugin under Source Code Management (e.g. the Git plugin):
Then click the Install button.
When this is completed, click the Continue as admin button on the following screen (instead of filling out Create First Admin User):
Note that, the next time you log in, your username will be admin and your password will be the one generated earlier.
On the next screen, click the button Start using Jenkins to complete the setup.
Scheduling a Run
To schedule test in Jenkins, start by clicking the New Item option on the left-hand side:
Provide a name for the new project, e.g. ProvarProject, then select the Freestyle project option and click the OK button.
If you are using version control, navigate to Source Code Management and select Git. (If you are not using version control, skip ahead to Build Triggers).
Enter the Repository URL, which can be copied by clicking the Clone or Download button against the repository.
Then provide Credentials (if applicable – this is only needed for private repositories):
To define the execution schedule, navigate to Build Triggers. Here you can define when you want Provar tests to be run. Choose Build periodically if you want to set up a schedule.
(Alternatively, you can also select Trigger builds remotely or Build after other projects are built if you want to trigger the execution using a script or following completion of other project builds.)
If you choose the Build periodically option, you will then need to enter an expression in Schedule to define when the tests should be run. This is done using a CRON expression:
To create this expression, you can write your own from scratch or copy or adapt one of the examples below (taken from here):
H H(3-4) * * *
Schedule: Every Day of the week, every Month in a year, every Day in a month at a time between 3 – 4 am in the morning at any minute.
H (30 -45) 3 * * *
Schedule: every Day of the week, every Month in a year, every Day in a month at 3 am in the morning between 30 -45 minutes.
*/5 * * * * :
Schedule: every 5 minutes.
0 8 * * *
Schedule: every day at 8h 00.
Enter your preferred syntax in Schedule, then check the validation message underneath to confirm the syntax is correct and the schedule has been defined as expected.
The ‘Would last have run at…’ information will show the most recent (past) run and next upcoming run, based on your syntax:
Finally, navigate to Build and click the Add build step dropdown, then choose Invoke ANT.
Leave Targets blank, unless you know that your Build File has multiple targets. (If your Build File was generated using the default settings supplied in Run Under ANT, leaving Targets blank will work fine.)
If you have modified the Build File so that there are multiple targets, enter the name of the desired target, e.g. runtests.
Then choose the Advanced… option and provide the build.xml path.
If the project is fetched from git, it should use the JENKINS_HOME variable (see screenshot above). If the project is local, it can be a filepath, e.g. C:\Users\Hetty\Documents\GitHub\dev\DevOps\ANT\build.xml:
Click the Save button.
Test your Build by clicking the Build Now option in the left-hand sidebar:
Build progress will be shown within Jenkins; if the build is successful, results will also be emailed as defined in the original Build File.
Configuring test reports
Once your build generates test results, Jenkins’ JUnit plugin can be used to display these results graphically:
When Provar tests are executed via Continuous Integration, Provar automatically generates a JUnit.xml file alongside its HTML and PDF reports. Using the JUnit plugin, this JUnit.xml file can be converted into a graphical visualization of the test results. The plugin also provides a web UI for viewing test reports and tracking failures.
Since Provar generates the JUnit.xml file automatically, very little further setup is required. One thing to keep in mind that the Results folder, which contains this JUnit.xml, should be generated or placed in Jenkins’ workspace folder.
For example: If the project is inside Jenkins’ workspace, define the results path as:
Follow the steps below to get the JUnit plugin up and running.
In Jenkins, click on the dropdown arrow on your Job Name and select Configure:
Navigate to ‘Post-build Actions’, then click the Add post-build action dropdown and select Publish JUnit test result report.
In Test report XMLs, set the path of the JUnit.xml file (note that this is generated inside your Results folder). This XML file should also be inside the workspace of the job of Jenkins. Enter the relative path of the results folder generated after execution:
Note that, before the first execution of your build, this field will show an error, since the file is not generated until after the execution.
Then save and trigger the job.
Once execution completes, you can view JUnit results by clicking on Test Result on the Execution View screen:
On the next screen, a History link is available to show the historical view or trend of execution. Note that historical data only be visible until two or more execution cycles.
This completes your Continuous Integration setup.
- General information
- Licensing Provar
- Provar trial guide and extensions
- Using Provar
- API testing
- Behavior-driven development
- Creating and importing projects
- Creating test cases
- Debugging tests
- Defining a namespace prefix on a connection
- Defining proxy settings
- Environment management
- Exporting test cases into a PDF
- Exporting test projects
- Managing test steps
- Namespace org testing
- Provar desktop
- Provar Test Builder
- Refresh and Recompile
- Reload Org Cache
- Running tests
- Searching Provar with find usages
- Secrets management and encryption
- Setup and teardown test cases
- Tags and Service Level Agreements (SLAs)
- Test cycles
- Test plans
- Testing browser options
- Tooltip testing
- Using the Test Palette
- Test Palette introduction
- Control test steps
- List compare
- Read test step
- String test steps
- UI Test Steps
- Using custom APIs
- Callable tests
- Data-driven testing
- Page objects
- Applications testing
- Database testing
- Document testing
- Email testing
- Mobile testing
- OrchestraCMS Testing
- Salesforce CPQ testing
- ServiceMax testing
- Skuid testing
- Vlocity testing
- Webservices testing
- Introduction to test scheduling
- Apache Ant
- Configuration for sending emails via the Provar Command Line Interface (CLI)
- Continuous integration
- Azure DevOps
- Running a Provar CI task in Azure DevOps
- Configuring the Provar secrets password in Microsoft Azure Pipelines
- Parallel execution in Microsoft Azure Pipelines using multiple build.xml files
- Parallel execution in Microsoft Azure Pipelines using targets
- Parallel execution in Microsoft Azure Pipelines using Test Plans
- Bitbucket Pipelines
- Circle CI
- GitLab CI
- Travis CI
- Parallel Execution
- Running Provar on Linux
- Salesforce DX
- Team foundation server
- Version control
- Zephyr Cloud and Server (Beta)
- Salesforce testing
- Adding a Salesforce connection
- Assert Page Error Messages on Add/Edit Product
- Internationalization support
- List and table testing
- Salesforce Release Updates
- Salesforce Lightning Testing
- Salesforce Lightning Web Component (LWC) locator support
- Service Cloud console testing
- Visualforce Testing
- Testing best practices
- Configurations and permissions
- Error messages
- Licensing, installation and firewalls
- Test Builder and test cases
- Release notes