Predictive Test Selection

How a machine learning model is trained

Every time tests run, your changes and test results are passed to Launchable to continuously train a model.

Model training looks at the changes associated with each build and which tests failed when tests ran. It builds associations between changed files and which tests tend to fail. It can be thought of as an advanced frequency counting algorithm that tracks associations between failures and source code.

Calculating test prioritization

One way to think about how Launchable prioritizes your tests is that with each successful test, Launchable's confidence grows that the entire run will be successful. The ideal algorithm optimizes for yielding the highest confidence as early as possible.

So confidence and individual test run time are the two primary determining factors for test prioritization.

Confidence is a function of the probability of failure for each individual test as tests run. Tests with a high probability of failure yield a higher confidence gain when successful. When tests with a low probability of failure pass, they yield smaller confidence gains.

Since the goal is to deliver as much confidence as quickly as possible, it makes sense for Launchable to deprioritize a long-running test if the confidence gain from that single test is not high enough to offset the gain of running shorter tests during the same period of time. This is exactly what the Launchable algorithm does.

For example, if test T8 has a high probability of failure and takes 3 minutes to run, and test T4 has a slightly lower probability of failure but only takes 300 milliseconds, Launchable will prioritize the shorter test (T4) before the longer test (T8) because it yields a higher confidence gain in a shorter period of time.

Subsetting Tests

If your tests take a very long time to run, you should consider running a subset of your tests earlier in your development cycle. We call this use case "shift-left." For example, if a test suite that runs after every pull request is merged takes 5 hours to run, a 30-minute version of the same suite could be run on every git push while the pull request is still open.

While you could accomplish this by manually selecting tests to run, this has the disadvantage that the tests most relevant to the changes present in a build may not be run until much later in the development cycle.

Launchable provides the ability to create a subset based on the changes present in the build every time you run tests. We call this a dynamic subset because the subset adapts to your changes in real-time.

A dynamic subset prioritizes all of your tests and then returns the first part of the total sequence for your test runner to run. The cutoff point can be based on either the maximum length of time you specify (e.g. 30 minutes in the above example) or the minimum confidence level you wish to achieve.

Launchable also provides the ability to run the rest of the tests that were not selected in your dynamic subset in a separate run.