Using Provar With Amazon Web Services (AWS) Device Farm
If you develop new mobile-optimized applications, you know that the testing process throughout the development lifecycle is exponentially more challenging given the wide range of mobile devices that your end-users may be operating to access your app.
That’s why more leading organizations are turning to the AWS Device Farm to help streamline the testing process by helping you test your application on a shared fleet of 2500+ devices remotely in the cloud. (We get it. The cube next to you probably isn’t big enough to charge that many phones.)
With Provar, you can create a wide range of testing scenarios and then rapidly reapply those tests by integrating those tests with the AWS Device Farm. (Side note, we also support testing via a number of different remote web and mobile testing platforms such as Sauce Labs, Perfecto, BrowserStack too.)
If you’d like to use Provar in conjunction with the AWS Device Farm, you can use the following information to run Provar test cases on a variety of Android devices (that use a Chrome browser) within the AWS Device Farm. To learn more, you can also check out the AWS Device Farm documentation that is available on the Salesforce website.
Prerequisites (a.k.a. Things you need)
- An AWS console account.
- A Provar ANT zip file.
- A Provar project with at least one test case.
Setting up your folder structure to execute test cases in the Device Farm
Since the AWS Device Farm is designed to conceal the webdriver, you will need to upload the project in an AWS Device Farm in a zip format.
For that, you will need to create a folder structure using the steps below:
(1) Create a folder (e.g. ProvarProject).
(2) Add the project folder, the Provar_ANT folder and the license file in the ProvarProject folder. Rename the license file with the name ProvarLicense.properties.
(3) In the project folder, there are test cases in the tests folder, build.xml, ANT folder and the .testproject file containing settings for Appium.
(4) Using ANT build.xml, execute your test cases.
(5) The Provar_ANT folder contains all of your required jar files.
(6) Create a zip file of ProvarProject.
Configuring your settings
You can configure the settings in a couple of different ways. You can either use the Provar UI or update the build.xml file and .testproject file.
(1) In the build.xml, a few changes are required.
Change provar.home where the Provar_ANT directory is available.
<property name="provar.home" value="../../Provar_ANT"/>
Change ProviderName (same as mentioned in .testproject) and DeviceName (any name) in the Run-Test-Case tag.
webBrowserProviderName="SampleProviderForDevice" webBrowserDeviceName="Google Pixel 2"
(2) In the .testproject file, select the browserProvider (same as ProviderName in build.xml). The values mentioned in PINK are mandatory fields and should have the same value as shown below.
<browserProvider name="SampleProviderForDevice" type="Appium"> <devices> <browserProviderDevice name="Google Pixel 2" platform="ANDROID"> <properties> <browserPlatformProperty name="deviceName" type="DesiredCapability" value="Google Pixel 2"/> <browserPlatformProperty name="platformName" type="DesiredCapability" value="ANDROID"/> <browserPlatformProperty name="platformVersion" type="DesiredCapability" value="9"/> <browserPlatformProperty name="udid" type="DesiredCapability" value="deviceUDID"/> </properties> </browserProviderDevice> </devices> <properties> <browserPlatformProperty name="gridUrl" value="http://127.0.0.1:4723/wd/hub"/> <browserPlatformProperty name="newCommandTimeout" type="DesiredCapability" value="100"/> </properties> </browserProvider>
ote: deviceUDID parameter will be replaced by actual device UDID at run time.
How to execute test cases
(1) Sign into your AWS console and navigate to the AWS Device Farm.
(2) Create a project in the AWS Device Farm console.
(3) In automated tests, select Create a new run button.
(4) Choose the web application and provide a name and click Next Step.
(5) In Configure Steps, select the Appium Node.js test type.
(6) Click on the Upload button and upload the ProvarProject.zip file.
(7) Once the project is uploaded, specify spec file. See the spec file below.
version: 0.1 # Phases are collection of commands that get executed on Device Farm. phases: # The install phase includes commands that install dependencies that your tests use. # Default dependencies for testing frameworks supported on Device Farm are already installed. install: commands: # By default, Appium server version used is 1.7.2. # You can switch to an alternate supported version from 1.6.5, 1.7.1, 1.7.2, 1.8.0 , 1.8.1, 1.9.1 by using a command like "avm 1.7.1" # OR # To install a newer version of Appium use the following commands: - export APPIUM_VERSION=1.9.1 - avm $APPIUM_VERSION - ln -s /usr/local/avm/versions/$APPIUM_VERSION/node_modules/.bin/appium /usr/local/avm/versions/$APPIUM_VERSION/node_modules/appium/bin/appium.js - java -version - ant -version - export PROVAR_HOME=$DEVICEFARM_TEST_PACKAGE_PATH - cd $DEVICEFARM_TEST_PACKAGE_PATH - mkdir -p /home/device-farm/Provar/.licenses - mv $DEVICEFARM_TEST_PACKAGE_PATH/ProvarProject/ProvarLicense.properties /home/device-farm/Provar/.licenses # The pre-test phase includes commands that setup your test environment. pre_test: commands: # We recommend starting appium server process in the background using the command below. # Appium server log will go to $DEVICEFARM_LOG_DIR directory. # The environment variables below will be auto-populated during run time. - echo "Start appium server" - >- appium --log-timestamp --browser-name Chrome --device-name $DEVICEFARM_DEVICE_NAME --platform-name $DEVICEFARM_DEVICE_PLATFORM_NAME --udid $DEVICEFARM_DEVICE_UDID --chromedriver-executable $DEVICEFARM_CHROMEDRIVER_EXECUTABLE >> $DEVICEFARM_LOG_DIR/appiumlog.txt 2>&1 & - >- start_appium_timeout=0; while [ true ]; do if [ $start_appium_timeout -gt 60 ]; then echo "appium server never started in 60 seconds. Exiting"; exit 1; fi; grep -i "Appium REST http interface listener started on 0.0.0.0:4723" $DEVICEFARM_LOG_DIR/appiumlog.txt >> /dev/null 2>&1; if [ $? -eq 0 ]; then echo "Appium REST http interface listener started on 0.0.0.0:4723"; break; else echo "Waiting for appium server to start. Sleeping for 1 second"; sleep 1; start_appium_timeout=$((start_appium_timeout+1)); fi; done; # The test phase includes commands that run your test suite execution. test: commands: # Go into the root folder containing your source code - echo "Navigate to test source code" - cd $DEVICEFARM_TEST_PACKAGE_PATH - echo "Start Provar/Appium test" - sed -i "s/deviceUDID/$DEVICEFARM_DEVICE_UDID/g" ProvarProject/Project/.testproject # Enter the command below to start the tests . The comamnd should be similar to what you use to run the tests locally. - cd ProvarProject/Project/ANT - ant # The post test phase includes are commands that are run after your tests are executed. post_test: commands: # The artifacts phase lets you specify the location where your tests logs, device logs will be stored. # And also let you specify the location of your test logs and artifacts which you want to be collected by Device Farm. # These logs and artifacts will be available through ListArtifacts API in Device Farm. artifacts: # By default, Device Farm will collect your artifacts from following directories - $DEVICEFARM_LOG_DIR - $DEVICEFARM_TEST_PACKAGE_PATH/ProvarProject/Project/ANT/
8) Create a new device pool and add devices in it.
(9) After selecting Device Pool, click the Next Step button.
(10) Click the Confirm and Start Run button.
(11) Check the status of the test execution using the links shown below.
Getting those reports you need
The AWS Device Farm provides many options (like Video, Test Specification log, Files, Screenshots) to verify your test execution status. And, it stores your reports for up to 400 days. These reports include metadata, logs, screenshots, and performance data.
(1) Video – It records the execution session on each device.
(2) Suites – It provides different types of status. Also, you can get the execution timings at suite level.
(3) Log files.
- General information
- Licensing Provar
- Provar trial guide and extensions
- Using Provar
- API testing
- Behavior-driven development
- Creating and importing projects
- Creating test cases
- Debugging tests
- Defining a namespace prefix on a connection
- Defining proxy settings
- Environment management
- Exporting test cases into a PDF
- Exporting test projects
- Managing test steps
- Namespace org testing
- Provar desktop
- Provar Test Builder
- Refresh and Recompile
- Reload Org Cache
- Running tests
- Searching Provar with find usages
- Secrets management and encryption
- Setup and teardown test cases
- Tags and Service Level Agreements (SLAs)
- Test cycles
- Test plans
- Testing browser options
- Tooltip testing
- Using the Test Palette
- Test Palette introduction
- Control test steps
- List compare
- Read test step
- String test steps
- UI Test Steps
- Using custom APIs
- Callable tests
- Data-driven testing
- Page objects
- Applications testing
- Database testing
- Document testing
- Email testing
- Mobile testing
- OrchestraCMS Testing
- Salesforce CPQ testing
- ServiceMax testing
- Skuid testing
- Vlocity testing
- Webservices testing
- Introduction to test scheduling
- Apache Ant
- Configuration for sending emails via the Provar Command Line Interface (CLI)
- Continuous integration
- Azure DevOps
- Running a Provar CI task in Azure DevOps
- Configuring the Provar secrets password in Microsoft Azure Pipelines
- Parallel execution in Microsoft Azure Pipelines using multiple build.xml files
- Parallel execution in Microsoft Azure Pipelines using targets
- Parallel execution in Microsoft Azure Pipelines using Test Plans
- Bitbucket Pipelines
- Circle CI
- GitLab CI
- Travis CI
- Parallel Execution
- Running Provar on Linux
- Salesforce DX
- Team foundation server
- Version control
- Zephyr Cloud and Server (Beta)
- Salesforce testing
- Adding a Salesforce connection
- Assert Page Error Messages on Add/Edit Product
- Internationalization support
- List and table testing
- Salesforce Release Updates
- Salesforce Lightning Testing
- Salesforce Lightning Web Component (LWC) locator support
- Service Cloud console testing
- Visualforce Testing
- Testing best practices
- Configurations and permissions
- Error messages
- Licensing, installation and firewalls
- Test Builder and test cases
- Release notes