It's common for teams to split up a large test suite into many smaller groups, often executed in parallel.
Perhaps your pipeline tests multiple components or plugins, or maybe your framework forces this kind of organization (e.g., dotnet test organizes tests by
.dll). In any case, all the groups are tested together (comprising a single Test Session), but each group has its own small test suite.
To better support this scenario, Zero Input Subsetting makes it possible to request a single "high level" subset across all components.
Now, a new concept called groups intends to improve the usability of this approach:
First, you can now use
--split-by-groupto split the "high level" subset file into one file for each group, simplifying test distribution across groups.
Second, when you use
--split-by-group, the CLI writes a special file informing you which groups you can skip entirely, saving setup time
Currently, only the Maven profile supports test groups, so this document uses instructions for Maven.
Before you can use
--split-by-group, you need to assign your tests to their respective groups. Each test can belong to one test group at a time. A group aligns with component/plugin/DLL -- whatever your organizational container is.
You can assign a set of tests to a group by running
launchable record tests with the
This means you'll run launchable record tests ten times if you have ten groups.
For example, we have three components: A, B, and C. Each group has 5 test items. We'll assign each to groups. (Note the use of
--group on each
launchable record tests)
For clarity, here are the commands:
# before building softwarelaunchable record build \--name jenkins-build-123\[...other options]...[build steps]...# before running tests create a test session so we can collect all the results togetherlaunchable record session \--build jenkins-build-123 > session-id.txt# componentA tests...[run componentA tests]...launchable record tests \--session $(cat session-id.txt) \--group=componentA \[...other options] \/path/to/componentA/results# componentB tests...[run componentB tests]...launchable record tests \--session $(cat session-id.txt) \--group=componentB \[...other options] \/path/to/componentB/results# componentB tests...[run componentC tests]...launchable record tests \--session $(cat session-id.txt) \--group=componentC \[...other options] \/path/to/componentC/results
The examples on this page describe a scenario with only 3 groups. This is just for illustrative purposes. In reality, this approach is for teams with lots of groups (e.g. 10+).
Once you've assigned your tests to groups, you can create a high-level subset and split it by group. This involves two commands run one after another in your main CI pipeline before you run any component pipelines:
launchable subsetwith the
--splitoption added. This option modifies the command's output to return a subset ID string instead of the subset contents. You'll use this ID in the next command.
Launchable creates a "high-level" exclusion list in this step and stores it for retrieval in the next step.
launchable split-subsetwith the
--split-by-groupoption. This command outputs several files for your pipeline (see below).
--subset-idoption is also required. This uses the value from the previous command.
In this step, Launchable splits the just-created exclusion list by group.
When you run
launchable split-subset with the
--split-by-group and the
--output-exclusion-rules option, the CLI creates several files:
Since you used
launchable subset, this file contains a list of the groups you can skip entirely.
subset-[groupname].txt(one file for each group)
Each file contains the normal subset output but only for that group's tests. You can pass these files into the test process for each group.
Since you used
launchable subset, these files contain exclusion rules. You're supposed to exclude these tests.
This file contains tests that had no group assignment, if there are any.
See the CLI reference for additional options.
In this example, we'll continue the scenario from above. We have three groups, each with five tests. We've already assigned each test to its respective group. Now we want to use that.
This diagram shows the flow. First, we create a subset from all the tests across all groups. Then we split those into groups. Note the special file
subset-groups.txt, which shows us we can skip component B entirely.
Note that the diagram shows the contents of
subset-component*.txt as a list of classes. This is the output format for Maven.
If you use a different test runner, your output might be different. Every test runner has its own exclusion syntax.
# before building softwarelaunchable record build \--name jenkins-build-123\[...other options]...[build steps]...# before running tests create a test session so we can collect all the results togetherlaunchable record session \--build jenkins-build-123 > session-id.txt# create the server side subsetlaunchable subset \--session $(cat session-id.txt) \--split \--get-tests-from-previous-sessions \--output-exclusion-rules > subset-id.txt# split that for use locallylaunchable split-subset \--subset-id $(cat subset-id.txt) \--split-by-groups \--output-exclusion-rules \maven
At this stage, we have several files:
Exclusion rules for component A
Exclusion rules for component B
Exclusion rules for component C
componentB, we can write a script to skip that group's test setup entirely. (How you do this depends on your setup. Need help? Let us know.)
Finally, we pass each component's exclusion file into the remaining test processes for each group:
subset-componentA.txtgets passed into the test process for component A
subset-componentC.txtgets passed into the test process for component C
We follow the normal instructions for using an exclusion rule (see the documentation for your test runner) so that only those tests run.
For example, here's a basic invocation of Maven for component A, complete with test recording at the end:
# component A# run component A tests with Launchable exclusionsmvn test -Dsurefire.excludesFile=$PWD/subset-componentA.txt# record testslaunchable record tests \--session $(cat session.txt) \--group=componentA \[...other options] \/path/to/componentA/results
As a result, we entirely eliminated component B (in this example) from the test process, saving time!
Because component B was entirely skipped, if there are any new group B tests, they didn't get run. This is the tradeoff of saving the setup time. To mitigate this, you should have a full run scheduled later in your pipeline, so they'll run then.