CircleCI is popular among Android developers for several reasons: it’s quick to get started, fast to execute your builds with high parallelism, (whether native, cross- or multi-platform), and even supports running Android emulators right from CircleCI with our Android machine images.

This article will show you how to build and test Android applications for an example project on the CircleCI platform. The full source code is available on GitHub - CircleCI-Public/android-testing-circleci-examples, and you can find the working pipelines in the corresponding CircleCI project.

To get the most out of this article you should have

  • Working knowledge of Android development
  • Familiarity with Android tooling and testing frameworks
  • Ability to use the Gradle build system

What’s available in CircleCI for Android

We have made a few tools available for you for building Android:

  • Docker image
  • Android machine image
  • Android orb

Docker image

The Docker image is the main executor teams use. It runs in a Docker container and so is very fast to start, and contains everything you might need to get started building Android projects. The full list of preinstalled tools can be found in the documentation. The images also ship with variants that have Node.js, the Android NDK, and browser tools preinstalled.

Machine image

The biggest change is the new Android machine image. This is a virtual machine image, so takes a bit more time than the Docker image mentioned above, but it contains all the tools you may need to build your Android applications, including the SDK, Google Cloud tools for Firebase, Python, Node.JS, and Fastlane. The biggest change however is the support for nested virtualisation, which allows you to run Android emulators from within this machine image.

Android machine images support nested virtualization, which allow running the x86 emulators in CircleCI jobs. If you have been developing for Android since the early years of the platform, you may remember that emulators were very slow. Then x86 emulators started supporting Intel’s Hyper-V hypervisors, which lets the emulator use the resources of the host computer directly. Using resource directly like this makes the emulators able to run much, much more smoothly and quickly.

For a long time, support was confined to our local hardware. Limited support on CircleCI slowed down emulators and made it difficult to run UI tests. Nested virtualization solves this problem by adding support for fast emulation to a massively reproducible CI/CD environment, making all the benefits of that available.

Android Orb

Finally there is the CircleCI Android orb which provides easy access to both images as executors, as well as handy jobs and commands for installing SDKs, running emulators, caching, testing, and more. You can read more about what the orb offers in the orb docs at this link.

Introducing the Android CI demo project

I have created a demo project, which is a fork of Google’s own Android Testing Codelab sources. The Codelab is a simple TO-DO list manager application and contains a combination of unit tests and instrumentation tests, with and without the UI. The project is also available to view on CircleCI. If you open it in Android Studio, make sure to switch to the Project view. Project view lets you view the .circleci/config.yml file we will be reviewing.

The project has a single app module, with the default debug and release variants. The test directory contains unit tests, while the androidTest directory contains instrumentation tests. The full CircleCI configuration is included, and I will be referring to it in this article. The configuration is located in .circleci/config.yml.

Building and testing CI for the sample Android app

Our CI process must do three things:

  1. Run unit tests inside a virtual machine
  2. Run instrumentation tests on the emulator
  3. Build a release version of the app

We will run unit tests and instrumentation tests side by side. If both types of tests are successful, we can assemble the release build. We will also add a condition to continue on to the release build only if new work has been commited to the “main” branch, and only following successful emulator tests on every Android version from 23 to 30. When our build meets this condition, we will have solid assurance that our application will work on many different Android devices.

Note: As I mentioned before, everything from this point on will refer to the CircleCI configuration file in .circleci/config.yml.

Using the CircleCI Android orb

The first step for creating our CI pipeline is to use the Android orb, which contains many of the steps already pre-written for you. The latest version is available in the orb docs.

After the orb is defined, the script specifies these jobs:

  • unit-test
  • android-test
  • release-build

One workflow, test-and-build, is also included.

version: 2.1

orbs:
  android: "2.1.2"

jobs:
  unit-test:
    ...
  android-test:
    ...
  release-build:
    ...
    # We'll get to this later

workflows:
  test-and-build:
    ...
  # We'll get to this later

Running unit tests

Our unit-test job contains an executor from the Android orb: android/android-docker. As mentioned this runs it in a Docker container which is extremely fast to start up.

In the steps stanza, there are a few things to note. First, we run android/restore-gradle-cache Gradle cache helps store our dependencies and build artifacts. After running the test command, we also run android/save-gradle-cache and android/save-build-cache to make sure subsequent builds run faster by using the cache. These all come from the Android orb, as indicated by the prefix android/.

We could also modify what gets cached by passing the find-args parameter to restore-gradle-cache. A great advanced example can be found in this open source project.

The main build step happens in the android/run-tests command, where we call ./gradlew testDebug within a test-command parameter. This command runs all unit tests for the debug build variant. As a bonus, this command comes with a default retry value to help you avoid test flakiness.

jobs:
  unit-test:
    executor:
      name: android/android-docker
      tag: 2022.08.1
    steps:
      - checkout
      - android/restore-gradle-cache
      - android/run-tests:
          test-command: ./gradlew testDebug
      - android/save-gradle-cache
      ...

Running Android tests on the emulator

Our android-test job is the one that uses the new Android emulator capability. The job uses the same Android machine executor as the build job. All subsequent jobs will use the same executor, and the cache restoration steps are also the same.

Emulator requires us to run on a machine image, so we specified android/android-machine as the executor. To improve build times, we specify the resource-class as xlarge. You can also run it on a large executor, but I have found that the lack of resources causes slower build times. I recommend that you do some experimentation on your own to find the right size executor for your project.

The next step is even shorter; we need just the android/start-emulator-and-run-tests command. This command does exactly what it says: it starts the specified emulator and runs the tests. We pass in the test-command again as the parameter (it uses the android/run-tests command). The system-image is the fully qualified Android emulator system image. We currently bundle our Android machine images back to SDK version 23. You could install more by using the sdkmanager tool.

jobs:
  android-test:
    executor:
      name: android/android-machine
      resource-class: xlarge
    steps:
      - checkout
      - android/start-emulator-and-run-tests:
          test-command: ./gradlew connectedDebugAndroidTest
          system-image: system-images;android-30;google_apis;x86
      ...

You could code all the emulator setup steps manually using the commands specified in the Android orb: create-avd, start-emulator, wait-for-emulator, and run-tests.

Storing the test results

There is more to testing than just running the tests, as we can get a lot of value from seeing exactly what has failed from right inside the CircleCI dashboard. For that we can use store_test_results functions built into the CircleCI platform, which will show our passing (or failing) builds.

The steps differ slightly between unit tests and instrumentation tests, as each of their respective Gradle tasks stores the tests in a different place. For unit tests the tests will be in app/build/test-results, and for instrumentation tests they will be in app/build/outputs/androidTest-results.

We suggest you create a new Save test results step, which uses the find utility and copies all test results files into a common location - like test-results/junit, and then store it from there. Also make sure to add the when: always parameter to the step, so the step runs regardless of whether the tests above succeed or fail.

Storing unit tests

For this project, we want the XML based results. Calling store_artifacts makes those results parseable on the CircleCI platform. You can also store the HTML output if you prefer.

...
- android/run-tests:
  test-command: ./gradlew testDebug
- run:
    name: Save test results
    command: |
        mkdir -p ~/test-results/junit/
        find . -type f -regex ".*/build/test-results/.*xml" -exec cp {} ~/test-results/junit/ \;
    when: always
- store_test_results:
    path: ~/test-results
- store_artifacts:
    path: ~/test-results/junit

Storing instrumentation tests

There are different regex command for different test types. The rest of the steps to store test results are identical to the unit tests example.

- android/start-emulator-and-run-tests:
        test-command: ./gradlew connectedDebugAndroidTest
        system-image: << parameters.system-image >>
    - run:
        name: Save test results
        command: |
          mkdir -p ~/test-results/junit/
          find . -type f -regex ".*/build/outputs/androidTest-results/.*xml" -exec cp {} ~/test-results/junit/ \;
        when: always
    - store_test_results:
        path: ~/test-results
    - store_artifacts:
        path: ~/test-results/junit

Making a release build and storing the artifacts

For this project, we need to store the application we are building. To complete this part of the process, we use the release-build job, which has two parts of note. A run step calls ./gradlew assembleRelease. Then, store_artifacts points to where the built APK is. For a full CI/CD project, you could sign and upload that APK to a beta distribution service. You could even create a release on Play Store with Fastlane. Both of those activties are out of scope for this article, though.

jobs:
  ...
  release-build:
    executor:
      name: android/android-machine
      resource-class: xlarge
    steps:
      - checkout
      - android/restore-gradle-cache
      - android/restore-build-cache
      - run:
          name: Assemble release build
          command: |
            ./gradlew assembleRelease
      - store_artifacts:
          path: app/build/outputs/apk/release

Running tests on multiple Android versions simultaneously

The Android platform evolves all the time, and each version has its own bugs and pecularities. I am sure you have encountered some of them. A good way to avoid undefined behaviour across different versions of Android is to test on as many of them as you can. The CircleCI job matrix feature makes it easier to run jobs on multiple versions.

To use the job matrix, add a parameter to the job you want to test on multiple versions. Our android-test job does that using the system-image parameter. We have also specified a default value for the parameter, so you can run the job without it.

To use the parameter, specify where you need its value by placing it within pairs of angle brackets - << >>:

 android-test:
    parameters: # this is a parameter
      system-image:
        type: string
        default: system-images;android-30;google_apis;x86
    executor:
      name: android/android-machine
      resource-class: xlarge
    steps:
      - checkout
      - android/start-emulator-and-run-tests:
          test-command: ./gradlew connectedDebugAndroidTest
          system-image: << parameters.system-image >> # parameter being used

To pass values to the parameter in the workflows, use the matrix parameter of the job we are calling.

workflows:
  jobs:
  ...
  - android-test:
      matrix:
        alias: android-test-all
        parameters:
          system-image:
            - system-images;android-30;google_apis;x86
            - system-images;android-29;google_apis;x86
            - system-images;android-28;google_apis;x86
            - system-images;android-27;google_apis;x86
            - system-images;android-26;google_apis;x86
            - system-images;android-26;google_apis;x86
            - system-images;android-26;google_apis;x86
            - system-images;android-26;google_apis;x86
      name: android-test-<<matrix.system-image>>

Making each run faster by caching

Alongside building our application, installing dependencies, and running tests, we often want to leverage cache as well. This allows us to do repeated tasks less often. Android and especially Gradle provides an excellent mechanism for this - the Gradle Build cache.

This lets you skip pre-built parts of the application.

Choosing what runs when

During development, we create many commits, and developers should be encouraged to push them to a remote repository as often as possible. Each one of these committs creates a new CI/CD build, though, and we might not need every test run on all possible devices for a single commit. On the other hand, if the main branch is our primary source of truth, we want to run as many tests on that as possible, to ensure it is always working as intended. We might want to initate the release deployment only on main, and not on any other branch.

We can use CircleCI’s requires and filters stanzas to define workflows that fit our process. Our release-build job requires the unit-test jobs. That means the release-build is put in the queue until all the unit-test jobs have passed. Only then is the release-build job run.

We can use filters to set up logical rules for running a particular job only on a specific branch or git tag. For example, we have a filter for our main branch that runs the android-test with a full matrix of emulators. In another example, the release build triggers only on main, and only when its two required jobs have run successfully.

workflows:
  test-and-build:
    jobs:
      - unit-test
      - android-test: # Commits to any branch - skip matrix of devices
          filters:
            branches:
              ignore: main
      - android-test: # Commits to main branch only - run full matrix
          matrix:
            alias: android-test-all
            parameters:
              system-image:
                - system-images;android-30;google_apis;x86
                - system-images;android-29;google_apis;x86
                - system-images;android-28;google_apis;x86
                - system-images;android-27;google_apis;x86
                - system-images;android-26;google_apis;x86
                - system-images;android-25;google_apis;x86
                - system-images;android-24;google_apis;x86
                - system-images;android-23;google_apis;x86
          name: android-test-<<matrix.system-image>>
          filters:
            branches:
              only: main 
      - release-build:
          requires:
            - unit-test
            - android-test-all
          filters:
            branches:
              only: main # Commits to main branch

Improving your Android application CI process

The steps I have described in this article are only a beginning. There are many ways you can improve the flow. Here are just a handful of ideas:

Build once, run many times

If you are an experienced Android developer you know that the connectedAndroidTest will always run the entire build process from the start. Using CircleCI, we could build the entire application and tests once, pass the artifacts down to following jobs, and simply run the tests on the emulators. This process could potentially save several build minutes per job run.

To make this happen, add three commandline steps in each emulator run: install the app, install the instrumentation app, and run the instrumentation tests. For our application, we would enter:

adb install app/build/outputs/apk/debug/app-debug.apk  
adb install app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk  
adb shell am instrument -w com.example.android.architecture.blueprints.reactive.test/androidx.test.runner.AndroidJUnitRunner

It is true that this code sends output to the the command line. That is not the tidy test output that Gradle provides. If you are motivated to make the output look better, this StackOverflow topic might help you.

Running on real devices

Emulators are one tool, but real world devices are often (sadly) much different. There is nothing that can replace a real-world device test. For that you can use a solution offering real devices in the cloud, like Sauce Labs or Firebase Test Lab.

Deploying to beta testers, and releasing the application

This build and test example project is just one part of a wider CI/CD story. You can take your CI/CD process much further. For example, you can automatically release new versions of your applications right to the Play Store. CircleCI has a handful of guides to help you. This tutorial is a great place to start. The official Fastlane documentation is another helpful resource.

Conclusion

In this article, I described building an Android application with CircleCI, and testing it. For our sample project, we used unit tests and a matrix of Android platform emulators, using the new CircleCI Android orb and machine images. We also touched on how to store and display test results, and how to orchestrate workflows to run tests will not give us the tidy test output that Gradle provides, and all output is on a portion of devices.

Do let me know if there is another Android topic you might want me to explore, either in an article or a live stream. Contact me on Twitter - @zmarkan.