Overview
Within Rocket Software, there is a 50 person team that delivers an analytics product based on the core product. We'll call them Rocket Insights. This team is a lighthouse within the greater organization, following cutting-edge development practices including tools like Bazel to perform build and test automation. Rocket Software looks to this team to lay the groundwork for wider improvements for developer productivity and happiness within the company.
With Launchable, Rocket Software can reduce test feedback time. We quantified the impact of slow feedback from tests (7.4k-10.2k developer hours per year). The result was that Launchable can reduce their build time by 40%.
Launchable can help this team within Rocket Software save up to $430k per year by reducing test feedback time to around 5.4k hours a year (a 40% reduction).
About Rocket Insights: A cutting edge team within a Future 50 software company
Rocket Software selected a division with the company called Rocket Insights to pilot Launchable. You can think of Rocket Insights as a startup within a much larger company. Rocket Insights builds a product that provides analytics on top of the core product from Rocket Software.
The product is delivered on-premises. Thus, software is delivered on multiple operating systems with multiple product versions. There are a sizable number of tests. Developers usually work on multiple release lines at once, and the cognitive cost of switching between lines is high.
The team is distributed across three geographic locations. Software development and testing continue around the clock.
The key challenge: slow test feedback
At its heart, the Rocket Insights division is about delivering on-premises packaged software on multiple operating systems. The challenges they face with slow feedback are applicable to a wide range of teams. Rocket Insights is typical of a few Launchable beta partners who fit this mode of software development.
The quotes in this section are representative of issues within the larger company. However, we can still bring significant benefits even though Rocket Insights is a smaller team within Rocket Software.
Rocket Insights' primary challenge is slow feedback caused by long cycle times. Features often take multiple weeks to deliver.
“If things are slow and I’m not productive… these things [slow development times] make people dissatisfied. It’s not just about productivity, it’s about satisfaction which drives retention.”
Rocket Insights also has a complicating factor. They use Bazel to cache tests and drive build times down. Hardware bandwidth is at a premium for Rocket Insights.
We have honed in on the “pre-merge” test cycle for Rocket Insights. The build times are about 1-2 hours for this team, and that is considered fairly reasonable today.
The cost of slow feedback for pre-merge tests
We were able to help Rocket Insights quantify the impact of slow feedback from tests. Their team of 50 developers spends approximately 7.4k-10.2k hours waiting on tests each year.
The team runs 36 builds per day or between 170-230 builds per week. With this build time data, we were able to compute the daily and yearly hours spent waiting by all developers and the corresponding dollar impact.
On average, there is a wait time of 4 days for every day of work delivered by this team!
40% improvement in developer feedback time by testing faster & smarter
We analyzed builds, code changes, and test results to create what we call a "Confidence Curve" for Rocket Insights (shown below). This curve represents how quickly a developer finds that her changes have a problem. It shows the percentage of tests that Launchable needs to run on the x-axis to achieve the confidence level on the y-axis. You can think of confidence as the likelihood that a regression will be detected.
This is a 40% reduction in build time for developers!
Furthermore, Rocket Insights has the option to choose any number on the red curve to optimize between feedback time and confidence. For example:
| Test Execution Time | Confidence |
|---|---|
| 55 seconds (2.5%) | 85.5% |
| 2 minutes (10%) | 89.5% |
| 4.4 minutes (20%) | 91.8% |
| 11 minutes (50%) | 95.3% |
Implications of the reduced cycle time for the development team
It requires multiple patch sets to get a change in (the average number of patch sets is 4.9). The throughput of the organization is about 6 changes per day.
A developer can get feedback approximately twice as fast with Launchable (51m → 31m) and consequently can hope to deliver a feature in a day or two versus the multiple it takes today.
Using Bazel and Launchable together
As mentioned earlier, the team uses Bazel to optimize builds. Bazel subsets test executions and thus test execution varies. The worst-case scenario is around 36 minutes with an average of 22 minutes (left graph below).
The Launchable model (right graph below) works in conjunction with Bazel’s incremental test selection. We bring the worst case down to 3.6 minutes and average down to 2.2 minutes.
Thus, the ML-based test automation approach from Launchable works on top of smart test automation tools such as Bazel.
The bottom line impact: cost savings
There are two ways to think about the benefits to Rocket Insights – bottom and top-line impact.
Here, we’ve focused on the bottom-line impact, which is cost savings in terms of time spent waiting because it is easier to quantify. However, we posit that the bigger impact is on the top-line because faster cycle times with higher quality shippable code translate to more benefits delivered by Rocket Insights to its customers
The bottom line impact is $427k saved each year.
This impact is achieved by saving developer hours waiting for feedback, or saving hours wasted by developers doing something other than their primary task. We think of this as “leaks” or “waste” in an organization that has always been hard to quantify, let alone get a handle on.
A massive drop in costs waiting for results
A 40% drop in test runs drives a corresponding drop in the yearly wait time. This was 3,400 hours saved (mean) for 50 engineers.
Saving 40% time implies a 40% dollar savings or roughly ~$427k. This saving easily exceeds the stated goal from the team as they began their engagement with us.
If we are able to save 30-60 minutes per developer per day, the savings will be staggering!
The top-line impact: test faster to deliver more customer-facing value
For Rocket Software, the top-line impact of using Launchable is that they are able to ship features faster at higher quality. We reduce software delivery cycle times because we can identify the right tests to verify changes. Developers getting faster feedback means their eventual changes are higher quality. Consequently, developers can push through smaller changes faster and more often. The ultimate winner is the customer, who now gets more value, earlier.
Happy developers
Wait times from tests is one of the biggest hurdles in a developer's day. Long wait times mean longer days or even longer weekends at work.
This is where Launchable shines! Shorter test cycle times without as much context switching implies developers can pick up a task and continue working on it without interruption until it is done. This brings employee satisfaction up. Ultimately, this helps engineering leaders drive down employee attrition. This is a potential benefit of bringing Launchable in for every company.