CLI reference

Getting started

Requirements

• Python 3.5 or newer

• Java 8 or newer

Install

The Launchable CLI is a Python3 package that can be installed via pip:

pip3 install --user --upgrade launchable~=1.0

This creates a ~/.local/bin/launchable executable that should be in your PATH. (See PEP-370 for further details.)

Authenticate

Set your API key:

export LAUNCHABLE_TOKEN=your_API_key

Verify

Then run launchable verify in your CI environment to see if you've successfully configured the CLI. If it succeeds, you'll see a message like the one below. If you see an error message, see Troubleshooting.

$ launchable verify

Organization: <organization>
Workspace: <workspace>
Proxy: None
Platform: 'macOS-12.0.1-x86_64-i386-64bit'
Python version: '3.9.9'
Java command: 'java'
launchable version: '1.34.0'
Your CLI configuration is successfully verified 🎉

Commands

inspect subset

Display the details of a subset request. See Subsetting your test runs for more info.

launchable inspect subset --subset-id 26876

You can use launchable inspect subset to inspect the details of a specific subset, including rank and expected duration. This is useful for verifying that you passed the correct tests or test directory path(s) into launchable subset.

The output from launchable subset includes a tip to run launchable inspect subset:

$ launchable subset --build 123 --confidence 90% minitest test/*.rb > subset.txt

< summary table >

Run `launchable inspect subset --subset-id 26876` to view full subset details

Running that command will output a table containing a row for each test including:

• Rank/order

• Test identifier

• Whether the test was included in the subset

• Launchable's estimated duration for the test

  • Tests with a duration of .001 seconds were not recognized by Launchable

inspect tests

Display the details of a record tests command. See Sending data to Launchable for more info.

launchable inspect tests --test-session-id 209575

record commit

Sends commit details to Launchable. Records multiple commits from repo(s).

launchable record commit --source ./src

Commit collection happens automatically as a part of record build, so normally this command need not be invoked separately. It's only used for Multiple repositories built/deployed separately then tested together (e.g. microservices).

`--import-git-log-output` option

Related to Running under restricted networks.

If the --import-git-log-output option is used, instead of reading the commits from the repository specified by --source, it reads the specified file for the commit data. The input file should contain the output of this Git command:

git log --pretty='format:{"commit": "%H", "parents": "%P", "authorEmail": "%ae", "authorTime": "%aI", "committerEmail": "%ce", "committerTime": "%cI"}' --numstat

record build

Creates a record of a Build in Launchable.

launchable record build [OPTIONS]

The act of recording a build teaches Launchable that the specified set of commits have turned into a build, and that this build is henceforth identified by the given name. This forms the basis of how Launchable calculates the changes.

record session

Creates a record of a test session in Launchable.

launchable record session [OPTIONS]

This command tells Launchable that you are about to begin testing a build that was been recorded earlier with the record build command. This is only needed in more complex scenarios.

The command outputs a string that you can save for use in other commands (like launchable subset and launchable record tests) in lieu of --build. We suggest saving the value either to an environment variable or to a text file:

# environment variable

launchable record build --name BUILD_NAME [OPTIONS]
export LAUNCHABLE_SESSION=$(launchable record session --build BUILD_NAME)
<run tests>
launchable record tests --session $LAUNCHABLE_SESSION [OPTIONS]

# text file

launchable record build --name BUILD_NAME [OPTIONS]
launchable record session --build BUILD_NAME > launchable-session.txt
<run tests>
launchable record tests --session $(cat launchable-session.txt) [OPTIONS]

(Otherwise, the command will write out a session ID to ~/.config/launchable/sessions/{hash}.txt. This location may change in the future, so don't rely on it.)

record tests

Send test results for the test session to Launchable.

launchable record tests [OPTIONS] TESTRUNNER ...

This command reads JUnit (or similar) XML report files produced by test runners and sends them to Launchable.

Exactly how this command generates the subset and what's required to do this depends on test runners. For available supported TESTRUNNER, see Integrations

split-subset

Splits an existing subset from Launchable into chunks. Related to Replacing static parallel suites with a dynamic parallel subset and Using groups to split subsets.

launchable split-subset [OPTIONS] TESTRUNNER ...

Intended for use with launchable subset with the --split option.

Options for Replacing static parallel suites with a dynamic parallel subset

--split-by-group outputs

When you run launchable split-subset with the --split-by-group option, the CLI creates several files:

• subset-groups.txt

  • By default, this file contains a list of the groups that you need to set up.

  • If --output-exclusion-rules was used with launchable subset, this file contains a list of the groups that you can skip entirely.

• subset-[groupname].txt (one file for each group)

  • Each file contains the normal subset output, but only for that group's tests. You can pass these files into the test process for each group.

  • By default, these files contain inclusion rules. If --output-exclusion-rules was used with launchable subset, these files will contain exclusion rules. You're supposed to exclude these tests.

• subset-nogroup.txt

  • This file contains tests that had no group assignment, if there are any.

--split-by-group-with-rest outputs

When you run launchable split-subset with the --split-by-group-with-rest option, the CLI creates several files:

• All of the files listed above under --split-by-group outputs

• rest-groups.txt

  • By default, this file contains a list of the groups that you don't need to set up.

  • If --output-exclusion-rules was used with launchable subset, this file contains a list of the groups that you can't skip.

• rest-[groupname].txt (one file for each group)

  • Each file contains the normal --rest (see #subset) output, but only for that group's tests.

  • By default, these files contain inclusion rules. If --output-exclusion-rules was used with launchable subset, these files will contain exclusion rules.

• rest-nogroup.txt

  • This file contains --rest tests that had no group assignment, if there are any.

stats test-sessions

Retrieves statistics about test sessions

launchable stat test-sessions

Output example:

{"averageDurationSeconds":653.168192926045,"count":311,"days":7}

Output is in JSON format:

subset

Produces a subset of tests to pass to your test runner.

launchable subset [OPTIONS] TESTRUNNER ...

Output is in JSON format:

Exactly how this command generates the subset and what's required to do this depends on test runners. For available supported TESTRUNNERs, see Integrations.

When none of --target, --time, and --confidence is specified, Launchable chooses the subset target. This is convenient on the initial setup when you are not sure what the subset size should be. Later, you can choose the right target after you see the statistics of your test suite and potential time-savings based on the Launchable accumulated data.

verify

Verify that the CLI can communicate with the Launchable service and that you're authenticated properly.

launchable verify

In order to avoid disrupting your CI/test process, the Launchable CLI is designed to tolerate & recover from service disruptions and other recoverable error conditions by falling back to no-op. This is an intentional design, but the downside is that such transparent failures can make troubleshooting difficult.

Therefore, we recommend you keep launchable verify || true in a recognizable spot in your CI process. This way, when you suspect a problem in Launchable, you can check the output of this command as a starting point.

Global options

--dry-run

The dry-run mode does not actually send a payload to the server, and it is helpful to check the behavior. You can also see which APIs will be requested and their payload contents in the output.

The payload contents will be output as an audit log, so if the log level is higher than the audit level, it will be forced to be set to the audit level.

Strictly speaking, it does not mean that no requests will be sent at all, but GET requests with no payload data or side effects may be sent. This is because sometimes the response data from the GET request is needed to assemble subsequent requests.

--log-level

You can use the --log-level option to output extra information from each command.

--log-level audit is particularly useful if you want to see exactly what data gets passed to Launchable when you run CLI commands. For example:

% launchable --log-level audit record build --name 1234 --source src=.
Processed 1 commits
AUDIT:launchable:send request method:post path:/intake/organizations/launchableinc/workspaces/awilkes/builds headers:{'Content-Type': 'application/json'} args:{'data': b'{"buildNumber": "1234", "commitHashes": [{"repositoryName": "src", "commitHash": "45b2e6d9df8e0013334354f30df1978c8b4196f8"}]}', 'timeout': (5, 60)}

--plugins

You can use the --plugins option to tell the CLI where to look for custom profiles/plugins.

For example, if you have developed (or been provided) a custom profile file named myProfile.py, place that file in the directory of your choosing (e.g. ~/launchable-plugins) and use it like this:

launchable --plugins ~/launchable-plugins record tests --build $BUILD myProfile /path/to/reports

Since --plugins is a global option, make sure to place it right after launchable but before subset or record in your command.