How testing in the cloud delivers value to development teams

Testing is an integral part of the software development process. In fact it is one of the key ways development teams can better understand how applications function. Testing also prevents changes in the codebase that can affect other parts of the code, enabling you to measure the quality of the software and eliminate any errors before users can interact with it.

Most development teams use unit and integration tests to assess their software. This testing often happens on the local machines of developers and testers. While local tests are useful for focusing developers’ attention on the behavior of the software components, testing in the cloud can make the entire development team more effective and help your organization ship better software, faster.

Cloud testing uses continuous integration platforms to automatically test the integrity of the software on every change to the codebase. These platforms produce instant results and make them available to the entire organization, surfacing issues quickly and improving team collaboration. Cloud testing also provides access to a wide range of compute resources, allowing you to run your tests without any hardware constraints.

This article discusses why your team should be testing in the cloud. It explores the benefits of cloud testing and how the insights from testing data can significantly improve the overall software development process.

Benefits of testing in the cloud versus locally

Development teams benefit from any software testing, whether it is in the cloud or on local machines. But some benefits that the cloud offers exceed those offered by testing locally. Some of your new, cloud-based abilities:

• Reducing the number of bugs found and minimizing their severity
• Visualizing and quantifying test benefits
• Strengthening decision making
• Providing real-time insights
• Accelerating feedback

Reducing the number and severity of bugs

Testing in the cloud ensures that tests always run right after every commit to guarantee that no one slips bugs into the product.

Cloud-based continuous integration platforms can trigger a test suite after every commit on every source control branch and merge request. This gives code reviewers confidence that all tests have passed before accepting a merge request.

A foundational principle of continuous integration is that developers should make small commits frequently throughout the day. By making granular commits and running tests on every change, you have a better view of the status of the code and can act faster whenever tests fail. This gives you more control over the development process, makes your team more agile and responsive, and improves the overall quality of your code.

Visualizing and quantifying test benefits

Testing in the cloud makes the impact of testing easier to visualize and quantify.

Executed offline, testing seems like a hidden cost that consumes developer and tester time without a measurable return on investment. Continuous cloud testing, in contrast, makes it easy to run tests often, track how often testing prevents bugs from extending to production, and track how many tests you have added over time in response to new features and bug reports.

Strengthening decision making

Testing in the cloud helps you make better decisions. If you must focus and carry out more tests in a certain area in response to bug reports, you may want to work more carefully on that area and feature in the future. This helps teams tackle capacity-related issues and stay responsive to emerging bugs or challenges.

Providing real-time insights

With cloud testing, you can access resources for computing online and monitor tests in real-time environments. This offers increased mobility, helps teams respond to any capacity-related issues, and collects information on the performance and service availability of the software.

Accelerating feedback

Another benefit of cloud testing is that it yields data faster than testing locally, where resource constraints often mean you have to run tests one at a time.

With continuous cloud testing, you can run multiple tests in parallel, or simultaneously. This means you receive actionable signals from your tests sooner, increasing productivity and efficiency. You can easily capture the output from tests and use visualization tools like CircleCI’s Insights dashboard to improve the development process.

How to capture valuable insights from testing data

One of the most significant benefits to teams from testing in the cloud is the ability to quickly derive insights from testing data. The best way to obtain these insights is to track meaningful metrics and industry benchmarks such as those cited in the 2024 State of Software Delivery Report. Once you have captured this data, it is equally important to share it with the rest of your team by uploading test results to a central location that all team members can access.

Track meaningful metrics

There are four common metrics that you can use to measure your team’s performance against industry benchmarks:

• Throughput
• Success rate
• Duration
• Mean time to recovery (MTTR)

Below, we’ll look at what each of these metrics measures and how you can refine your development processes to optimize for each.

Throughput

Throughput is the average number of workflow runs per day. A developer’s update to the codebase in a shared repository activates a workflow.

If the goal for a development team is to increase throughput, making smaller commits more often can reduce the threat of errors and make it easier to mitigate them quickly. You should make this measurement as often as required.

Success rate

Your success rate should always be high (90 percent or above) for the primary branch of the product. This rate is the number of passing runs divided by the total number of runs in a period.

The ability to measure the success rate of current workflows is essential in establishing development targets. For example, feature branches, where developers experiment with new solutions to customer problems, are likely to have a much lower success rate than the main branch. That’s because main should always be the most recent stable version of the product. If the success rate on the main branch is too low, increase testing to catch bugs before they are committed to the main branch.

Duration

Duration is the length of time it takes for a workflow to run. A duration of ten minutes is a good target for the length of a standard workflow. With this duration, you can move faster and benefit from the information generated through the CI pipelines.

To improve your workflow duration, you can take advantage of time-saving features offered by CI/CD providers, such as dependency caching, Docker layer caching, and running your tests in parallel.

Mean time to recovery

Mean time to recovery (MTTR) is the average time it takes to achieve a passing build signal after a failed test. MTTR is the most important metric to track, as it measures how quickly your development team can recover whenever they encounter a failure. The ability to rapidly respond to bugs is a crucial part of staying competitive and delivering value to end users.

Ideally, your mean time to recovery should be less than 60 minutes. MTTR is directly tied to your workflow duration; you can’t fix bugs faster than you can test for them, after all. If your team is struggling to address bugs promptly, try to “fail faster” by making smaller, more incremental commits. This helps tests run faster and makes defects in the code easier to resolve.

Upload your test results to the cloud

Uploading all test results and metrics to a CI/CD platform makes it easy for your team — and the whole organization — to better understand the state of the product at every phase.

Many continuous integration platforms support storing your test results for a set period of time, allowing you to share your results with stakeholders and analyze trends in your test data. With CircleCI, you can use the store_test_results step to upload data from your test runs.The data is then available in the Tests tab of the web app for up to 30 days. For long-term storage of your test results, you can also use the store_artifacts step to store test results as a shareable artifact.

More information on the benefits of storing your continuous integration test results is available in this video:

Uploading your test results not only gives you a centralized location for all team members to analyze and review test data, but it also makes it easier to access it with data visualization tools. Some platforms offer built-in dashboards for analyzing your test runs. CircleCI’s Insights dashboard, for example, provides detailed data on important metrics like workflow duration and success rate. And, Insights can automatically identify flaky tests to help improve the reliability of your test suite.

Many platforms also offer integrations with tools such as Datadog, Honeycomb, Katalon, PractiTest, and others. These tools can provide custom reports on how your team is performing relative to important benchmarks.

How to apply testing insights to improve your team’s development process

Use these testing insights to improve your development process. Remember, data is only valuable if converted into practical insights.

Good testing may identify high-value user journeys or simulate errors to test the system’s response. With the insights gathered from testing, you can document those high-value journeys, develop processes to handle simulated errors, and stay ahead of your user’s needs.

Your team should make it a habit to review testing data, summarize the insights, make recommendations, and share the results with other teams that can benefit from it. How this looks depends on the structure of your team; a larger team might have a dedicated tester who does it. On smaller teams, the responsibility may rotate among team members on a sprint-by-sprint basis.

Conclusion

Testing in the cloud is valuable — not just for the success of your application, but for your whole development workflow. If you are already using cloud-based CI, the platform you use will already have built-in test runners, meaning it costs very little to get started and start reaping the benefits.

The key to capturing the output from your tests and making effective changes is receiving detailed insights. The Insights dashboard makes CircleCI a good place to start your cloud testing. With Insights, CircleCI helps you monitor and optimize your workflows, and helps you make informed decisions about your development and testing process.

To learn more about how you can use CircleCI to gain valuable, actionable insights into your development workflows, get started with a free plan today.