Combine Pytest and Launchable for faster test feedback

The Launchable CLI provides a simple, uniform interface between your CI process and the Launchable service.

To make integration even easier, we’ve built profiles into the CLI for popular build and testing tools. These profiles abstract away the nuances of specific languages and test runners so you only have to add a few lines to your build script to get started.

One of these profiles is for the popular Python testing tool Pytest. In this post, we’ll show you how to connect Launchable with Pytest to train a machine learning model and get faster test feedback.

What is Launchable?

Launchable uses machine learning to identify the most important tests to run for a specific code change. This capability lets you run a smaller set of important tests earlier and more frequently in your software development lifecycle, providing faster feedback to developers without compromising quality.

To enable this, you can use the Launchable CLI to connect Launchable to your test runner both to train a machine learning model and to get and run test recommendations supplied by Launchable.

Connecting Launchable and Pytest using the Launchable CLI

After installing the Launchable CLI package from PyPI and setting your API key, you’ll need to add three Launchable commands to your build/test pipeline.

First, you need to record a build to tell Launchable the code changes that are being tested. Launchable uses this information to train the machine learning model and to recommend tests to run for those specific changes.

To do this, you add the `record build` command to your CI script. You give the build a name and point the CLI to your Git repository on the CI server (`.`):

launchable record build --name <BUILD NAME> --source src=.

Then, you need to record test results after running tests to train the machine learning model:

launchable record tests --build <BUILD NAME> pytest ./test-results/

You'll notice that this command uses the same `<BUILD NAME>` value; that's how Launchable learns that these test results relate to the new commits in the build.

Once you’ve started sending test results to train a model, you can start subsetting tests. Over time, the test recommendations will improve as the model learns.

Before running tests, you need to request a dynamic subset of tests from Launchable and pass those to Pytest to run them.

You provide the same build name, a target confidence percentage populate the subset, and your test directory:

launchable subset \
  --build <BUILD NAME> \
  --confidence <TARGET> \
  pytest tests > launchable-subset.txt

The subset command outputs a list of recommended tests specifically for that buildto launchable-subset.txt which you then pass into Pytest:

pytest --junit-xml=test-results/subset.xml $(cat launchable-subset.txt)

See docs.launchableinc.com for more documentation and examples.

Now you can run a subset of a long running test suite more often. For example, you could run a subset of long running end-to-end UI tests on every git push instead of only after every merge. Or you could subset your pull request tests to get faster feedback, earlier.