Use-cases for Predictive Test Selection
Use-case 1: In-place subset of a test suite
Challenge: An existing test suite may be taking a long time to run. Some teams' capacity for executing tests is finite & limited, yet the demand to run tests is too high. Even in cases where test execution capacity is scalable & elastic, the number of tests are so many that it’s costing too much money.
To that end, the team would want to shorten the execution time of that test suite.
This use-case can also serve to enhance developer productivity, especially when merging changes. Test failures arising from flaky tests often lead to frustration and unnecessary delays.
Proposed Solution: Introduce a subset in place. Run a subset of tests at the same stage of your software delivery lifecycle
Value unlocked by the use-case
Reduction in test runtime: The Time Savings Report gives the amount of time saved running a test suite.
Lower machine costs (cloud compute costs, mobile devices, cheaper UI testing with Selenium etc)
More frequent runs: Since the test runtime reduces, your team can run tests more frequently and test more often (if so desired).
Optimized resource allocation (instead of running tests that don’t fail, use your testing budget to run tests that fail)
Implementing the solution
This is the easiest use-case to implement with Launchable. A Launchable subset is used to replace the current full test execution.
Adding a defensive run: After subsetting your tests, you should make sure to run the full suite of tests at some point later in your pipeline. This enables you to catch any issues that may not be caught by the subset.
For example, once you start running a subset of an integration test suite that runs on pull requests, you should make sure to run the full integration test suite after a PR is merged (and record the outcome of those runs with launchable record tests).
Use-case 2: Shift-left a test suite to find issues earlier in the pipeline
Challenge: The feedback on changes may be coming in too late because the tests are towards the right in your delivery pipeline (e.g. UI tests, E2E, Nightly tests. These are run infrequently because of long run times). In several teams, another common challenge is that their main/dev branch is too unstable, causing QA people a lot of overhead to deal with failures.
Proposed Solution Run a subset of tests earlier in your software delivery lifecycle ("shift left")
Value unlocked by the use-case
Ensure a stable main/dev branch, maintain release readiness, and reduce effort in tracking failures.
Faster feedback time by running tests run earlier (e.g. a nightly test run can be run on every PR, or every hour to get fast feedback; reducing feedback to developers by hours if not days)
More testing by more frequent test runs
Optimized resource allocation (instead of running tests that don’t fail, use your testing budget to run tests that fail and run them more often)
Implementing the solution
If you aim to run a short subset of a long test suite earlier in the development process, you may need to set up a new pipeline to run tests in that development phase. For example, if you currently run a long nightly test suite and want to run a subset of that suite every hour, you may need to create a pipeline to build, deploy, and run the subset if one doesn't exist already.
You'll also want to continue running the full test suite every night (and recording the outcome of those runs with launchable record tests).
