Zero Input Subsetting

Normally, when you run launchable subset, the Launchable CLI gathers the full list of tests on the client side and submits it with the subset request. (Highlighted in gray)

The subset request then returns a list of tests to include (i.e., run these tests):

Diagram of subsetting using Predictive Test Selection

We've created a complementary approach called Zero Input Subsetting. With this approach, the CLI does not have to gather and submit the full list of tests. Instead, the server generates the full list of tests from the last two weeks of recorded sessions. To ensure new tests are run, the CLI outputs exclusion rules instead of inclusion rules.

Zero Input Subsetting works better with some Test Session Layouts than others, so contact your Customer Success Manager before you start using this feature. We're here to help!

You can adopt this approach by adding two options to launchable subset:

--get-tests-from-previous-sessions, and
--output-exclusion-rules

The subset request then returns a list of tests to exclude (i.e., don't run these tests):

The following CLI profiles/integrations support Zero Input Subsetting:

Let us know if you want to see support for another test runner!

Also, see Using groups to split subsets which expands this behavior.

Instructions for test runners/build tools

Android Compatibility Test Suite (CTS)

Find your run cts command in your CI script. These commands will go before that command.

First, request an exclusion pattern:

launchable subset --get-tests-from-previous-sessions --output-exclusion-rules --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> cts > launchable_exclusion_filter.txt

See the CLI reference for details about <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

Then pass that exclusion pattern into run cts using xargs:

cat ./launchable_exclusion_filter.txt | xargs ./tools/cts-tradefed run cts

dotnet test + NUnit

Adding Launchable NUnit integration as a dependency

Your test project needs to depend on Launchable NUnit integration. Run the following command against your project:

dotnet add package Launchable.NUnit

You then need to add the assembly level attribute to activate this integration in your test project. Typically this goes into AssemblyInfo.cs:

// activate Launchable NUnit integration
[assembly: Launchable.NUnit.Launchable]

See https://github.com/launchableinc/nunit for full source code.

Request and execute subset

Find your dotnet test command in your CI script. These commands will go before that command.

First, request an exclusion pattern:

launchable subset --get-tests-from-previous-sessions --output-exclusion-rules --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> dotnet --bare > launchable_exclusion.txt

See the CLI reference for details about <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

Then set the LAUNCHABLE_REST_FILE_PATH enviroment variable to point this file, then run the tests like you normall do:

export LAUNCHABLE_REST_FILE_PATH=$PWD/launchable_exclusion.txt
dotnet test

Gradle

First, you'll request an exclusion list from your full test suite. Then, you'll pass this list to Gradle.

Requesting an exclusion list

First, you need to add a snippet to your Gradle config to enable test exclusion via the Gradle command line:

test {
    if (project.hasProperty('excludeTests')) {
        exclude project.property('excludeTests').split(',')
    }
}

Then, find the gradle command used to run tests in your CI script.

Before that command, run launchable subset to request an exclusion list. The subset and exclusion lists are generated from the union of tests recorded in the last two weeks.

launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> --get-tests-from-previous-sessions --output-exclusion-rules gradle > launchable-exclusion-list.txt

See the CLI reference for details about <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

This creates a file called launchable-exclusion-list.txt. This file contains a list of test classes formatted for passing into Gradle like this:

-PexcludeTests=com/example/FooTest.class,com/example/BarTest.class

Running a subset of tests

Then pass this file into your existing command, like shown below.

gradle test $(cat launchable-exclusion-list.txt)
# equivalent to gradle test -PexcludeTests=com/example/FooTest.class,com/example/BarTest.class

Note: If the exclusion list is very large, it may be unable to specify it directly from the command. In that case, you can change the Gradle config to read from launchable-exclusion-list.txt.

Change the Gradle config as follows:

test {
    if (project.hasProperty('excludeTestsTxt')) {
        exclude new File(project.property('excludeTestsTxt')).text.replaceFirst('-PexcludeTests=', '').trim().split(',')
    }
}

Then, specify the exclusion tests file from the command.

gradle test -PexcludeTestsTxt=$PWD/launchable-exclusion-list.txt

Summary

In summary, here's the flow before:

# your normal command to run tests looks something like this
gradle test <OPTIONS>

And the flow after:

# request an exclusion list from all tests
launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> --get-tests-from-previous-sessions --output-exclusion-rules gradle > launchable-exclusion-list.txt
# run tests, excluding deprioritized tests, leaving only the recommended subset
gradle test <OPTIONS> $(cat launchable-exclusion-list.txt)

Gradle + TestNG

First, you'll request an exclusion list from your full test suite. Then, you'll pass this list to Gradle.

Requesting an exclusion list

First, find the gradle command used to run tests in your CI script.

Before that command, run launchable subset to request an exclusion list. The subset and exclusion lists are generated from the union of tests recorded in the last two weeks.

launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> --get-tests-from-previous-sessions --output-exclusion-rules gradle --bare > launchable-exclusion-list.txt

See the CLI reference for details about <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.
Don't forget the --bare option after gradle!

This creates a file called launchable-exclusion-list.txt. This file contains a list of test classes formatted for passing into Gradle, like this:

com.example.FooTest
com.example.BarTest
...

Running a subset of tests

First, you need to add a dependency declaration to build.gradle so that the right tests get excluded when TestNG runs:

dependencies {
    ...
    testRuntime 'com.launchableinc:launchable-testng:1.2.1'
}

Then export the exclusion list file path as an environment variable before you run mvn test, like shown below.

export LAUNCHABLE_REST_FILE_PATH=$PWD/launchable-exclusion-list.txt
gradle test <OPTIONS>

Summary

In summary, here's the flow before:

# your normal command to run tests looks something like this
gradle test <OPTIONS>

And the flow after:

# request an exclusion list from all tests
launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> gradle --bare <PATH TO SOURCE> > launchable-exclusion-list.txt
# run tests, excluding deprioritized tests, leaving only the recommended subset
export LAUNCHABLE_REST_FILE_PATH=$PWD/launchable-exclusion-list.txt
gradle test <OPTIONS>

Maven

First, you'll request an exclusion list from your full test suite. Then, you'll pass this list to Maven.

Requesting an exclusion list

Find the mvn test command used to run tests in your CI script.

Before that command, run launchable subset to request an exclusion list. The subset and exclusion lists are generated from the union of tests recorded in the last two weeks.

launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> --get-tests-from-previous-sessions --output-exclusion-rules maven > launchable-exclusion-list.txt

See the CLI reference for details about <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

This creates a file called launchable-exclusion-list.txt that you can pass into Maven.

Running a subset of tests

To exclude deprioritized tests and only run the recommended subset, use the -Dsurefire.excludesFile option. For example:

mvn test <OPTIONS> -Dsurefire.excludesFile=$PWD/launchable-exclusion-list.txt

If your build already depends on surefire.includesFile, or <includes>/<includesFile>, those and our exclusion list will collide and not work as expected. Contact us to resolve this problem.

Summary

In summary, here's the flow before:

# your normal command to run tests looks something like this
mvn test <OPTIONS>

And the flow after:

# get an exclusion list from the server
launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> --get-tests-from-previous-sessions --output-exclusion-rules maven > launchable-exclusion-list.txt
# run tests, excluding deprioritized tests, leaving only the recommended subset
mvn test <OPTIONS> -Dsurefire.excludesFile=$PWD/launchable-exclusion-list.txt

Maven + TestNG

First, you'll request an exclusion list from your full test suite. Then, you'll pass this list to Maven.

Requesting an exclusion list

Find the mvn test command used to run tests in your CI script.

Before that command, run launchable subset to request an exclusion list. The subset and exclusion lists are generated from the union of tests recorded in the last two weeks.

launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> --get-tests-from-previous-sessions --output-exclusion-rules maven > launchable-exclusion-list.txt

See the CLI reference for details about <BUILD NAME> and <OPTIMIZATION TARGET OPTION>.

This creates a file called launchable-exclusion-list.txt that you can pass into Maven.

Running a subset of tests

First, modify your pom.xml so that it includes Launchable TestNG integration as a test scope dependency:

<dependency>
  <groupId>com.launchableinc</groupId>
  <artifactId>launchable-testng</artifactId>
  <version>1.2.1</version>
  <scope>test</scope>
</dependency>

Then export the exclusion list file path as an environment variable before you run mvn test, like shown below.

export LAUNCHABLE_REST_FILE_PATH=$PWD/launchable-exclusion-list.txt
mvn test <OPTIONS>

Summary

In summary, here's the flow before:

# your normal command to run tests looks something like this
mvn test <OPTIONS>

And the flow after:

# get an exclusion list from the server
launchable subset --build <BUILD NAME> <OPTIMIZATION TARGET OPTION> --get-tests-from-previous-sessions --output-exclusion-rules maven > launchable-exclusion-list.txt
# run tests, excluding deprioritized tests, leaving only the recommended subset
export LAUNCHABLE_REST_FILE_PATH=$PWD/launchable-exclusion-list.txt
mvn test <OPTIONS>

Last modified 1 year ago