Self-hosted versus cloud-based mobile app testing

Testing is a vital part of the mobile app development process. Your team can use testing to evaluate the quality, security, and reliability of mobile apps before releasing them to your users. Users who expect their applications to be highly performant and intuitive.

There are two ways DevOps teams can perform testing for mobile apps: on-premise (also called self-hosted) or in the cloud. But which of these is the best option for your team?

This article compares self-hosted and cloud-hosted mobile testing to help you decide on the best app testing option for your organization.

What is self-hosted mobile testing?

Self-hosted mobile testing is run on local machines set up at a dedicated physical location. It involves running the app in your developer test environment to check if its usability and functionality meet all requirements before releasing the app to the market.

With self-hosted testing, you’re responsible for setting up and maintaining servers and testing infrastructure. You must also monitor the machines and execute software upgrades. Creating the configurations and test environment for self-hosted mobile testing is time-consuming and often incurs high costs, but it does offer several benefits.

Self-hosted mobile testing lets you set up and closely monitor your tests. It also gives you more control over your configurations and of setting up software and hardware. Data remains on the premises, and security compliance is applied on the premises.

While not well-suited to every mobile application, self-hosted mobile testing is beneficial in some use cases. For example, a development company specializing in health care apps may consider building a testing solution in-house to ensure that the apps comply with regulations like HIPAA. These apps contain confidential information, and the tester must ensure they can safely hold the highest level of data. With a self-hosted testing solution, the developer maintains end-to-end control of the app, the data for testing, and the testing environment. This control makes it easier to address potential security concerns and enables developers to create tests that meet their requirements without involving third parties.

What is cloud-based mobile testing?

Cloud-based testing relies on third-party service vendors to provide cloud computing resources. These resources use a pay-per-use pricing model for your mobile application tests. The provider owns the test infrastructure, runs the tests, and delivers the results. Cloud-hosted testing occurs in real-time, so the results are immediately available.

Cloud-based testing is a virtual way of testing mobile applications. You can use real mobile devices hosted on the cloud or virtual software environments, such as emulators and simulators, to mimic the conditions of an actual mobile device. With emulators and simulators in a virtual environment, you can run tests faster without requiring an actual mobile device for testing.

The virtualization of cloud-based mobile testing allows you to allocate and scale resources without affecting the entire testing system. This differs from self-hosted testing, where you must expand your capacity to handle an increasing number of devices and make sure that newly added devices communicate with the rest of the infrastructure.

If you’re testing mobile apps at scale, you can’t rely on self-hosted sequential testing because it slows down the release cycle. Cloud-based testing allows you to test many configurations in a short amount of time — regardless of how expansive your application is.

For example, if you’re testing the UI functionality of an app, you can test it with many different configurations, operating systems, and environments in the cloud. You can run an unlimited number of tests simultaneously. Running tests at scale helps your team detect and fix bugs faster, reducing the time to market for your apps.

With a self-hosted mobile app testing solution, your testing abilities are limited by your infrastructure. It costs you time and money to run many tests on a self-hosted solution, and may require an infrastructure upgrade. The flexibility that cloud-based mobile testing provides lets you scale up your mobile application tests without any overhead.

Advantages of cloud-based mobile testing solutions

In addition to the advantages outlined above, there are many other benefits to using cloud-based mobile testing, including reduced overhead, flexible compute options, and faster test execution.

Low overhead

With cloud-based testing, you don’t need to find servers, procure licenses for testing tools and programs, or install them. The cloud provider gives you access to a virtual testing lab with a library of test management tools, execution tools, and operating systems. The virtual lab also contains middleware, storage, and everything else required to create a test environment that mirrors a physical mobile test environment.

Additionally, cloud-based testing tools save you from the hassle of updating or patching your test infrastructure to address security vulnerabilities or get access to the latest software versions. Cloud providers manage these processes for you, regularly deploying security updates and updating VM images so that you can stay focused on delivering new features to your users rather than maintaining complex infrastructure.

Variety of computing options

Cloud-based test platforms offer a range of computing resources, allowing you to customize your test execution environment and computing power to suit your team’s needs. This flexibility makes it easier to test your application across multiple platforms and software versions, giving you confidence that your application will perform as expected for your end users.

As your application and testing needs change, the flexible compute offerings in the cloud allow you to seamlessly scale up your available RAM and CPU to handle increased workloads. Some cloud providers also provide options to test in specialized environments and architectures like GPU and ARM, giving you additional flexibility to test your mobile applications in a production-like environment.

Speed

Cloud-based mobile app tests run faster than self-hosted tests because the cloud supports optimizations like caching, parallelization, and test splitting. Running tests faster helps developers find and fix bugs quickly with less resource spending.

• Caching allows you to reuse previously computed or retrieved data. A cache stores a subset of data transiently so that you don’t need to access the underlying slower storage layer for data retrieval. It’s quick and easy to read data from an in-memory cache, which improves the speed with which you test mobile apps.
• Parallelization is an automated process that mainly runs on virtual machines. A tester executes several tests simultaneously in multiple browser configurations and device combinations. Parallelization allows you to leverage virtualization and the cloud to perform more tests faster than you could if you were testing on your infrastructure. This process is resource intensive and better executed in the cloud, where you can request more resources when needed on a pay-per-price billing model. Parallel testing is usually automated and runs at high concurrency in the cloud, which means a lower cost.
• Test splitting allows you to execute tests simultaneously across multiple containers. If you have a test suite that contains dozens of tests, running tests sequentially would take a significant amount of time. But if you split these tests into multiple concurrent processes, you can receive feedback quickly and with less wait time.

You can learn more about the benefits of caching, parallelization, and test splitting in the following resources:

• How Lightspeed optimized iOS test runs with parallelism and caching
• Config best practices: dependency caching
• Config best practices: concurrency and parallelism
• A guide to test splitting

Conclusion

Self-hosted testing is a capital-intensive option for testing mobile apps. You have to spend on procuring hardware, software, and performing maintenance. In cloud-based testing, the infrastructure is already there, the testing environment is ready, and the service provider handles the maintenance.

The cloud is a highly scalable environment that allows you to level up and down testing resources based on your needs. While there are cases in which self-hosted mobile testing makes sense, most mobile app testing can be completed more efficiently and effectively in the cloud.

CircleCI provides many options for accelerating mobile development, including automating software build and release processes in a comprehensive CI/CD pipeline. To learn more about how CircleCI and cloud-based mobile testing can help you ship quality applications quickly, explore the benefits of continuous integration for mobile app development. When you are ready to optimize your mobile application testing experience with the speed and flexibility of cloud-based test automation, get started by signing up for a free CircleCI account.