pytest (Integration)

Details about how Launchable integrates with pytest.

This is a reference page. See Getting Started, Sending data to Launchable, and Predictive Test Selection for more comprehensive usage guidelines.

Launchable interfaces with pytest via

  1. the Launchable pytest plugin

  2. the Launchable CLI

Launchable pytest plugin

We offer a new way to integrate Launchable, a native pytest plugin.

Installing the plugin

The Launchable pytest plugin is a Python3 package that you can install from PyPI.

{% hint style="warning" %} The plugin requires Python 3.7+Pytest 4.2.0+and Java 8+. {% endhint %}

If you use Pipenv, you can install the plugin into your repository:

pipenv install --dev pytest-launchable

Or, you can install the CLI in your CI pipeline by adding this to the part of your CI script where you install dependencies:

pip3 install pytest-launchable

You don't need to install Lanchable CLI separately because the plugin automatically installs the CLI and uses it internally.

Setting your API key

First, create an API key for your workspace at app.launchableinc.com. This authentication token allows the pytest plugin to talk to Launchable.

Then, make this API key available as the LAUNCHABLE_TOKEN environment variable in your CI process. How you do this depends on your CI system:

CI system Docs
Azure DevOps Pipelines Set secret variables
Bitbucket Pipelines Variables and secrets
CircleCI Using Environment Variables
GitHub Actions How to configure a secret
GitLab CI GitLab CI/CD environment variables
GoCD Setting variables on an environment
Jenkins

Injecting secrets into builds

(Create a global "secret text" to use in your job)

Travis CI Environment Variables

Generate a config file

launchable-config is a command-line tool to generate and validate configuration files. The Launchable pytest plugin uses this config.

First, generate a new config file:

# via pipenv
pipenv run launchable-config --create
# via pip
launchable-config --create

This generates a template .launchable.d/config.yml file in the current directory that looks like this:

# Launchable test session configuration file
# See https://docs.launchableinc.com/resources/cli-reference for detailed usage of these options
#
schema-version: 1.0
build-name: commit_hash
record-build:
# Put your git repository location here
source: .
max_days: 30
record-session:
subset:
# mode is subset, subset-and-rest, or record-only
mode: subset
# you must specify one of target/confidence/time
# examples:
# target: 30% # Create a variable time-based subset of the given percentage. (0%-100%)
# confidence: 30% # Create a confidence-based subset of the given percentage. (0%-100%)
# time: 30m # Create a fixed time-based subset. Select the best set of tests that run within the given time bound. (e.g. 10m for 10 minutes, 2h30m for 2.5 hours, 1w3d for 7+3=10 days. )
target: 30%
record-tests:
# The test results are placed here in JUnit XML format
result_dir: launchable-test-result

You can then edit the config file per the directions below.

Recording test results (pytest plugin)

Update your config file

In .launchable.d/config.yml:

  1. Check that the source option in the record-build section points to your Git repository (the default is ., the current directory).

  2. Check that the mode option in the subset section is set to record-only

Verify your config file

Verify the contents of the .launchable.d/config.yml file:

# via pipenv
pipenv run launchable-config --verify
# via pip
launchable-config --verify

If any problems are reported, edit the file accordingly.

Use the plugin with pytest

Then, just add an --launchable option to the pytest command. It is very easy:

pytest --launchable <your-pytest-project>

If the configuration file is not in the current directory, use the --launchable-conf-path option:

pytest --launchable --launchable-conf-path <path-to-launchable-configuration-file> <your-pytest-project>

This will:

  1. Create a build in your Launchable workspace

  2. Run your tests

  3. Submit your test reports to Launchable

  4. Leave XML reports in the launchable-test-result by default

Requesting and running a subset of tests

In .launchable.d/config.yml:

  1. Check that the source option in the record-build section points to your Git repository (the default is ., the current directory).

  2. Check that the mode option in the subset section is set to subset or subset_and_rest based on your needs

  3. Check that one of the three optimization target options are set (targetconfidence, or time)

Verify your config file

Verify the contents of the .launchable.d/config.yml file:

# via pipenv
pipenv run launchable-config --verify
# via pip
launchable-config --verify

If any problems are reported, edit the file accordingly.

Use the plugin with pytest

Then, just add an --launchable option to the pytest command. It is very easy:

pytest --launchable <your-pytest-project>

If the configuration file is not in the current directory, use the --launchable-conf-path option:

pytest --launchable --launchable-conf-path <path-to-launchable-configuration-file> <your-pytest-project>

This will:

  1. Create a build in your Launchable workspace

  2. Request a subset of tests based on your optimization target

  3. Run those tests (or run all the tests if subset_and_rest mode is chosen)

  4. Submit your test reports to Launchable

  5. Leave XML reports in the launchable-test-result by default

Launchable CLI

See Recording test results with the Launchable CLI and Subsetting with the Launchable CLI for more information.