Speed up XCUITest execution with parallelism and test splitting

In this article, I’ll show you how to reduce the execution time of XCUITest (UI tests on iOS simulators) by splitting and running them in parallel.

Automated tests and CI/CD platforms like CircleCI are necessary for iOS application development. It is important not only to introduce them once but to improve them continuously.

When application code grows and automated tests increase, the execution time of build and test in CI/CD gets longer. Longer build and test time results in slower development speed.

Because UI tests like XCUITest (Xcode UITest) need to be run on actual iOS devices or iOS simulators, the execution time tends to be long.

CircleCI has a variety of ways to reduce execution time in iOS application development:

• Multiple resource classes for macOS VM
• Dependency caching
• Test splitting and parallelism

In this article, I’ll show you how to combine CircleCI’s test splitting and parallelism with fastlane to reduce execution time.

Prerequisites

For this tutorial, you will need:

• A CircleCI account, preferably the Performance plan, for optimal macOS VM concurrency.
• Xcode installed. This tutorial uses Xcode 13.4.1.
• fastlane installed.
• Familiarity with writing XCUITest (Xcode UITest).

How to add test splitting and parallelism to your XCUITest runs using CircleCI and fastlane (overview)

The sample code for the iOS application introduced in this tutorial is available on GitHub.

This includes multiple UI tests. The execution time is changed by inserting sleep(), as shown here:

import XCTest

class CircleCIDemoUITests5: XCTestCase {
    func testTapButton5() throws {
        // UI tests must launch the application that they test.
        let app = XCUIApplication()
        app.launch()
        
        // Elements
        let text = app.staticTexts["text"]
        let button = app.buttons["button"]
        
        XCTAssertEqual(text.label, "Hello, world!")
        sleep(50)
        button.tap()
        XCTAssertEqual(text.label, "Button Tapped!")
    }
}

To run all tests in series using fastlane, use the run_tests Action and write the following.

desc "Run all UITests"
lane :ui_test_all do
  run_tests(
    scheme: "CircleCIDemoUITests",
    devices: ["iPhone 13 (15.4)"]
  )
end

To run the tests in parallel, follow these steps:

1. Pre-build (build_for_testing) to run UI tests
2. Launch multiple macOS VM/iOS simulators
3. Split tests based on execution time
4. Run split tests in parallel
5. Upload test results which include execution time

Pre-build build_for_testing to run UI tests

All iOS application tests, including UI tests, must be built before running. For this tutorial, I did the building part in advance, so you can run the tests in parallel on multiple macOS VMs later on.

From Xcode8, build-for-testing and test-without-building allow you to separate build and test execution.

When you implement build-for-testing in fastlane, enable the build_for_testing parameter in the run_tests Action.

desc "Run all UITests"
lane :build_for_ui_test do
  run_tests(
    scheme: "CircleCIDemoUITests",
    devices: ["iPhone 13 (15.4)"],
    derived_data_path: "dist",
    build_for_testing: true
  )
end

The derived_data_path parameter allows you to specify the path to the artifact.

This is the CircleCI build_for_ui_test job:

build_for_ui_test:
  macos:
    xcode: 13.3.1
  resource_class: macos.x86.medium.gen2
  steps:
    - checkout
    - ruby/install-deps
    - run: bundle exec fastlane build_for_ui_test
    - persist_to_workspace:
        root: .
        paths:
          - dist

This includes the following steps:

• Install fastlane using the Ruby orb.
• Run fastlane (build_for_testing).
• Make pre-build artifacts available for the next job, persist_to_workspace.

Test splitting and parallelism of UI tests

After the pre-build build_for_ui_test job is completed, run the ui_test_parallel job for splitting and running UI tests in parallel.

ui_test_parallel:
  parallelism: 2
  macos:
    xcode: 13.3.1
  resource_class: macos.x86.medium.gen2
  steps:
    - checkout
    - macos/preboot-simulator:
        device: iPhone 13
        version: "15.4"
    - attach_workspace:
        at: .
    - ruby/install-deps

Launch multiple macOS VMs to run the tests in parallel. You can increase or decrease the number of macOS VMs running in parallel by setting the value of the parallelism key.

The steps in this job include:

• Pre-boot iOS simulator using the macOS Orb.
• Download pre-build artifacts using attach_workspace.
• Install fastlane using the Ruby orb.

Then, split the tests and run them in parallel.

Here is the fastlane to run:

desc "Run specific UITests"
lane :ui_test_without_building do |options|
  run_tests(
    scheme: "CircleCIDemoUITests",
    devices: ["iPhone 13 (15.4)"],
    only_testing: options[:tests],
    derived_data_path: "dist",
    test_without_building: true
  )
end

This code snippet uses the run_tests Action, but with two differences. First, it enables the test_without_building parameter. This time we have pre-built the app build-for-testing, so we can run the tests without building by enabling the test-without-building parameter.

Second, it enables the only_testing parameter. This parameter allows only certain tests, not all tests, to be run. The target of only_testing (specific tests) can be passed from outside.

Here’s the part of the CircleCI configuration file that splits and runs the tests in parallel:

- run:
    name: Split tests and run UITests
    command: |
      CLASSNAMES=$(circleci tests glob "CircleCIDemoUITests/*.swift" \
        | sed 's@/@.@g' \
        | sed 's/.swift//' \
        | circleci tests split --split-by=timings --timings-type=classname)
      FASTLANE_ARGS=$(echo $CLASSNAMES | sed -e 's/\./\//g' -e 's/ /,/g')
      bundle exec fastlane ui_test_without_building tests:$FASTLANE_ARGS
- store_test_results:
    path: fastlane/test_output/report.junit

The target test files are retrieved by circleci tests glob and split by circleci tests split.

You can see that --split-by=timings flag is added to circleci tests split. This takes the timing data from the previous test run to split a test suite as evenly as possible over a specified number of test environments running in parallel. The --split-by=timings flag gives the lowest possible test time for the compute power available. You can learn more about test splitting and parallelism in the following video.

In this case, five tests with different execution times were run in two parallel runs. There was little variation in the test execution time.

Conclusion

In this article, I introduced how to reduce the execution time of XCUITest (UI tests on iOS simulators) by splitting and running them in parallel.

Key take-aways for this solution:

• Timing-based test splitting gives you the lowest possible test time for the available compute power.
• Parallelism level can be easily scaled by setting the value of the parallelism key in your CircleCI config file.
• Even when running multiple macOS VMs in parallel, as in this case, cost performance is high.

Note: CircleCI charges by macOS VM usage (credits) per minute, not by parallel number. To learn more, review our CircleCI pricing and plan information.

If you want to improve the performance of iOS application development, I hope you will find this useful.