Orb Testing Methodologies

This guide covers various best practices for testing orbs.

Introduction

Orbs are open source and contributions are expected, therefore, as with all software, setting up a solid testing pipeline for your orb is important. Because orbs are created in YAML, it may seem difficult to test them effectively. With the orb development kit, there is a simple path to implementing a full range of robust tests for your orb.

Validation

The most basic forms of testing for orbs are configuration validation and code linting. When an orb is packed and published it must be both valid YAML, and valid CircleCI syntax. Both of these checks are automatically applied when using the orb development kit, through the CI/CD pipeline set out in the project’s config file at .circleci/config.yml. Config validation and code linting can also be performed manually, locally.

# Snippet from test-pack workflow
test-pack:
    unless: << pipeline.parameters.run-integration-tests >>
    jobs:
      - orb-tools/lint # Lint Yaml files
      - orb-tools/pack # Pack orb source + validate config
      - shellcheck/check:
          dir: ./src/scripts
          exclude: SC2148

When you first make a commit to your orb repository, the test-pack workflow is triggered, which contains several jobs related to validating and testing your orb.

Learn more about what is included in orb project config files in the Orb Publishing Process guide.

YAML lint

The first job listed within the workflow, orb-tools/lint, is from the orb-tools orb, which is a major component of the orb development kit. The orb-tools/lint job is responsible for basic YAML linting. You can modify the linting rules or other settings via the job’s parameters, which are listed on the orb registry.

Config validation

The test-pack workflow will run the orb-tools/pack job, in parallel with your YAML linting job (orb-tools/lint), to automatically pack and validate the configuration.

A singular orb.yml file (a packed orb) can be validated with the CircleCI CLI via circleci orb validate orb.yml. However, using the orb development kit, we rarely work out of singular YAML files. Instead, your configuration file is automatically validated after it has been packed with the circleci orb pack <dir> > orb.yml command.

Shellcheck

One of the major benefits of using the orb development kit is the ability to import external bash scripts into your final orb. Because you can keep your bash scripts in the src/scripts directory, you can run additional tests against your scripts.

The most basic tests to run against bash scripts are a form of validation: “shellchecking”. This is similar to a linter for Bash, you can find out more at shellcheck.net.

In the test-pack workflow, you will find the shellcheck orb is included. The shellcheck orb steps are completely optional and can be removed, especially, if your orb does not require scripts to be imported.

Unit testing

If you are taking advantage of the orb development kit’s <<include(file)>> file inclusion feature and src/scripts directory to store and source your bash files, you can write true integration tests for your scripts.

Unit testing BASH with BATS-Core

The test-pack workflow includes the bats/run job, which is responsible for automatically executing .bats tests within the src/tests directory.

When you initialize your orb, the greet.yml command is generated, which includes the greet.sh shell script. Also included, is a test case example using the BATS-Core (Bash Automation Testing System) framework, in the src/tests directory, named greet.bats


# Source: https://github.com/CircleCI-Public/Orb-Project-Template/blob/master/src/commands/greet.yml

description: >
  This command echos "Hello World" using file inclusion.
parameters:
  to:
    type: string
    default: "World"
    description: "Hello to whom?"
steps:
  - run:
      environment:
        PARAM_TO: <<parameters.to>>
      name: Hello Greeting
      command: <<include(scripts/greet.sh)>>

# Source: https://github.com/CircleCI-Public/Orb-Project-Template/blob/master/src/scripts/greet.sh

Greet() {
    echo Hello "${PARAM_TO}"
}

# Will not run if sourced for bats-core tests.
# View src/tests for more information.
ORB_TEST_ENV="bats-core"
if [ "${0#*$ORB_TEST_ENV}" == "$0" ]; then
    Greet
fi

# Source: https://github.com/CircleCI-Public/Orb-Project-Template/blob/master/src/tests/greet.bats

# Runs prior to every test
setup() {
    # Load our script file.
    source ./src/scripts/greet.sh
}

@test '1: Greet the world' {
    # Mock environment variables or functions by exporting them (after the script has been sourced)
    export PARAM_TO="World"
    # Capture the output of our "Greet" function
    result=$(Greet)
    [ "$result" == "Hello World" ]
}

BATS Core

The Bash Automation Testing System is an open source testing framework that provides a simple way to test UNIX programs.

Within your src/tests is a README with a full and updated tutorial for creating BATS test cases.

Each .bats file within the src/tests directory will be automatically loaded and tested by the bats/run job provided by the bats orb.

Example

For a real-life example, take a look at the tests in our Shellcheck orb.

Remember, including bats tests are optional and can be removed from your configuration file if desired.

Here is a simplified snippet from the Shellcheck orb’s BATS test suite.

# example BATS test
setup() {
    # Sourcing our bash script allows us to acces to functions defined within.
    source ./src/scripts/check.sh
    # Our script expects certain envrionment variables which would be set as parameters.
    # We can "mock" those inputs here.
    export SC_PARAM_OUTPUT="/tmp/shellcheck.log"
    export SC_PARAM_SHELL="bash"
}

teardown() {
    # Logs are recorded in each function.
    # We will echo it out on error, but otherwise remove it to indicate no issue.
    rm -rf /tmp/shellcheck.log
}

# This is a test case in the BATS framework.
# This is essentially just a function with a name.
# For each test case, setup() will run -> test -> teardown() -> repeat.

# Ensure Shellcheck is able to find the two included shell scripts
@test "1: Shellcheck test - Find both scripts" {
    # Mocking inputs
    export SC_PARAM_DIR="src/scripts"
    export SC_PARAM_SEVERITY="style"
    export SC_PARAM_EXCLUDE="SC2148,SC2038,SC2059"
    Set_SHELLCHECK_EXCLUDE_PARAM
    Run_ShellCheck
    # Test that 2 scripts were found
    [ $(wc -l tmp | awk '{print $1}') == 2 ]
    # If an error is thrown anywhere in this test case, it will be considered a failure.
    # We use a standard POSIX test command to  test the functionality of the "Run_ShellCheck" function.
}

Integration testing

Up until this point, all testing has happened prior to packing the orb, and is applied to the code itself, not the finalized functioning orb. For the final, critical part of orb testing, you will test your orb commands and jobs to ensure they work as intended in production. This happens after the validation tests have run and a new development version of your orb is published.

After the development version of your orb has been published, the integration-test_deploy workflow will be triggered automatically to test it.

The integration-test_deploy workflow runs a series of final integration tests, and if all pass, and you are on your main deployment branch, you can deploy your orb.

Testing orb commands

The first job you will see in the integration-test_deploy workflow is the integration-test-1 job, a sample integration test included with the hello-world orb, generated by the orb-init command.

You can see the definition of the integration-test-1 job above in the jobs key.

  integration-test-1:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - <orb-name>/greet

In your local version, <orb-name> will be replaced by the orb name you provided. This job offers a way for us to test our orb’s jobs in a real CircleCI environment.

Replace the steps of this job with commands from your orb. You could include a sample project if needed or otherwise just run your orb’s commands to ensure they do not result in a failure.

Testing orb jobs

If we needed to test our orb’s jobs, as well as commands, we can simply add our orb job right next to the integration-test-1 job in our config under the integration-test_deploy workflow.

integration-test_deploy:
    when: << pipeline.parameters.run-integration-tests >>
    jobs:
      - integration-test-1
      - my-orb/orb-job
      - orb-tools/dev-promote-prod-from-commit-subject:
          requires:
            - integration-test-1
            - my-orb/orb-job

What’s next?

Once you have added new orb features, and created passing tests, it is time to publish your orb to the Orb Registry. View the Orb Publishing Process guide for information on automatically publishing semantically versioned orbs.

See also

  • Refer to Orbs Concepts for high-level information about CircleCI orbs.
  • Refer to Orb Publishing Process for information about orbs that you may use in your workflows and jobs.
  • Refer to Orbs Reference for examples of reusable orbs, commands, parameters, and executors.
  • Refer to Configuration Cookbook for more detailed information about how you can use CircleCI orb recipes in your configurations.