Managing complex test session layouts
Instructions for properly recording data to Launchable when your build and test process spans several machines.
This page relates to Recording test results with the Launchable CLI.
The Test Session is one of Launchable's core concepts. When you record test results, those results are recorded to a test session. When you request a subset of tests from Launchable, the subset is linked to a test session, too. This concept is useful because tests might run several times against the same build; it helps disambiguate those runs.
Normally, the Launchable CLI manages session creation and usage in the background. However, if your build and test processes are split across multiple machines, or if your tests are parallelized across multiple machines, you'll need to create test sessions yourself.
Build and test processes happen on different machines
Normally, the Launchable CLI handles creating, saving, and retrieving a session ID in the background. When you run launchable subset
or launchable record tests
, the CLI checks for an existing file in ~/.launchable
. This file is written when you run launchable record build
.
However, if you need to record tests (launchable record tests
) or request a subset (launchable subset
) on a different machine than the one where launchable record build
ran, the ~/.launchable
file won't be present, and you'll need to manually create a test session using the launchable record session
command at the beginning of your test process.
This command outputs a string that you can store and then pass into the --session
option in launchable subset
and launchable record tests
.
Here's some pseudocode to illustrate:
# machine 1# build steplaunchable record build --name <BUILD NAME> [OPTIONS]# build softwarebundle install# machine 2# before running tests, save a session token# you'll use this token again laterlaunchable record session --build <BUILD NAME> > launchable-session.txt# get a subset, if applicablelaunchable subset --session $(cat launchable-session.txt) [OPTIONS]# run testsbundle exec rails test [OPTIONS]# machine 3# record test resultslaunchable record tests --session $(cat launchable-session.txt) [OPTIONS]
Combining test reports from multiple runs
Some pipelines execute multiple test runs against a build, outputting distinct test report(s) across several machines. Depending on your layout (see Test Session), you may want to combine these into a single test session. This section explains how to do this.
This may also be the case if you execute tests of a single type across several parallel runs, but usually the test runner can combine reports from parallel runs for consumption from a single place.
If all the test reports for a session can be collected from a single machine, you don't need to use this method.
To tie multiple launchable record tests
invocations to a single test session:
Use the
launchable record session
command to create a session IDPass this ID into the corresponding
--session
parameter inlaunchable record tests
(note: pseudocode):
## build step# before building software, send commit and build info# to Launchablelaunchable record build --build <BUILD NAME> [OPTIONS]# build software the way you normally do, for examplebundle install## test step# before running tests, save a session token# you'll use this token to group the test reports together laterlaunchable record session --build <BUILD NAME> > launchable-session.txt# start multiple test runs# machine 1# run testsbundle exec rails test# send test results to Launchable from machine 1# Note: You need to configure the line to always run whether test run succeeds/fails.# See each integration page.launchable record tests --session $(cat launchable-session.txt) [OPTIONS]# machine 2# run testsbundle exec rails test# send test results to Launchable from machine 2# Note: You need to configure the line to always run whether test run succeeds/fails.# See each integration page.launchable record tests --session $(cat launchable-session.txt) [OPTIONS]## repeat as needed...## finish multiple test runs
You can read more about launchable record session
in the CLI reference.