The nose plugin automatically sends test results to Launchable when you run nose with
However, if you want to submit test reports from a full test run to help train the model, run your tests with the
nosetests --launchable-build-number <BUILD NAME> \--launchable-record-only
First, install the Launchable plugin for nose using PIP:
$ pip install nose-launchable
To run a meaningful subset of tests, invoke
nosetests with three extra flags:
# subset tests with Launchablenosetests --launchable-build-number <BUILD NAME> \--launchable-subset \--launchable-subset-options <LAUNCHABLE CLI SUBSET OPTIONS>
--launchable-build-number flag tells Launchable which build is being tested. This should be the same value you used for
launchable record build before. The
--launchable-subset flag enables subsetting.
--launchable-subset-options flag tells the plugin which Launchable CLI subset options to use. For example, to use the Launchable CLI's
--target option, the flag should look like
--launchable-subset-options '--target 10%'. This runs a subset of the most useful test targets that will run in 10% of the full execution time. You can also use
--launchable-subset-options '--time 100' to run a subset of the most useful test targets that will run in 100 seconds. See the CLI reference for more options.
--launchable-subset-target PERCENTAGE option is still available; it functions the same as
--launchable-subset-options '--target PERCENTAGE'. To accommodate future CLI options, however, we recommend using
--launchable-subset-options where convenient.
Some teams manually split their test suites into several "bins" to run them in parallel. This presents a challenge adopting Launchable, because you don't want to lose the benefit of parallelization. Luckily, with split subsets you can replace your manually selected bins with automatically populated bins from a Launchable subset.
For example, let's say you currently run ~35 minutes of tests split coarsely into four bins and run in parallel across four workers:
Worker 1: ~20 minutes of tests
Worker 2: ~15 minutes of tests
With a split subset, you can call Launchable once in each worker to get the bin of tests from a subset for that runner.
The high level flow is:
Manually record a build (see Recording builds)
Manually record a test session against that build and store the value returned from Launchable for use later
In each nosetests invocation, request the bin of tests that worker should run. To do this, run
--launchable-test-sessionoption set to the session ID value you saved earlier, and
# main$ launchable record build --name 12345 --source src=.$ launchable record session --build 12345builds/12345/test_sessions/678910# machine 1$ nosetests --launchable-subset --launchable-test-session builds/12345/test_sessions/678910 --launchable-subset-options "--target 40% --bin 1/2"# machine 2$ nosetests --launchable-subset --launchable-test-session builds/12345/test_sessions/678910 --launchable-subset-options "--target 40% --bin 2/2"
If you encounter issues running nose with the Launchable plugin, you can set the
LAUNCHABLE_DEBUG environment variable to
1 before running tests to print debug logs.
# enable debug logsexport LAUNCHABLE_DEBUG=1# run tests with Launchablenosetests --launchable-subset ...
The Launchable nose plugin is open source and available on GitHub. Pull requests are always appreciated!