Creating automated build, test, and deploy workflows for orbs, part 2

In my last post, I discussed how to set up an ideal, automated CI/CD process for a new orb. The goals of this process are version control, multiple levels of testing, and automated deployment. Now, let’s walk through an example. I’ll use our Azure CLI orb, published just last month, as it is small and simple enough to be fairly easily understood, yet also a bit complex, as it does integrate with Microsoft Azure, a third-party cloud provider.

The config.yml file for an orb built, tested, and deployed with this process is not that complicated. Our Orb Tools orb has abstracted out most of the basic mechanics of automated orb development. The Azure CLI orb’s config.yml contains two distinct workflows. One workflow for basic validation and dev publishing, and another for integration/usage testing and possible orb deployment to production:

workflows:
  lint_pack-validate_publish-dev:

  # …

  integration_tests-prod_deploy:

This setup allows you to do usage and integration testing of an orb within a single config.yml file, in a single repository, tied to a single git commit. It allows for bypassing some alternative orb-testing methods which can be hackier or more cumbersome (although by no means ineffective), such as inlining your orb, using an external repository for testing, or evaluating your new orb source in what is essentially a local job, running on CircleCI, using the machine executor.

By default, all workflows defined in a config.yml file run simultaneously. However, we’ve set these two up so that one runs on every commit and the other is triggered only by git tags. By controlling how these git tags are created, we can ensure that our integration testing workflow runs only after we’ve done our basic linting/validation and published the dev release of our orb that we want to further test.

Workflow 1:

lint_pack-validate_publish-dev

Our first workflow is entirely orb-ified! That is, you don’t have to do anything to use it in your own orb repository’s configuration files other than call a series of jobs defined in our Orb Tools orb and pass them the requisite parameters. Let’s quickly go through these jobs, one by one:

• Job 1: orb-tools/lint

  - orb-tools/lint:

The orb-tools/lint job will lint all the YAML files in a given directory using the yamllint CLI tool, as packaged by the singapore/lint-condo Docker image. You can provide a custom .yamllint configuration file, or use some basic defaults included in the job.

• Job 2: orb-tools/pack

  - orb-tools/pack:
      requires:
        - orb-tools/lint

orb-tools/pack is made for folks writing orbs in destructured YAML format, which I highly recommend using! For any orb with more than one or two separate commands or jobs, organizing your orb source this way will make your code much easier to parse, debug, and develop. It’s akin to breaking a monolith into smaller, more modular services, or writing React components instead of a single HTML file cluttered with HTML, JavaScript, and CSS code.

The design of orbs, utilizing a few higher-level object types (i.e., commands, executors, jobs, examples), lends itself quite intuitively to a filesystem tree in which each type has its own folder, and each folder contains all instances of that type, each in their own YAML file. A single @orb.yml file serves as the “entrypoint” to this system, containing version information and a description. Other orbs referenced by your orb can also be put here, or each can be given its own file within an orbs directory, as well (since orbs can reference other orbs). Taking a look at the /src directory of our Azure CLI orb example should make clear what this all tends to look like.

By default, orb-tools/pack will do the following:

1. Checkout your project
2. Pack your orb source folder into a single orb.yml file
3. Validate this new orb.yml file (circleci orb validate orb.yml)
4. Persist the orb.yml file to a workspace, so it can be used in downstream jobs

As you can see from the YAML snippet above, the job has defaults such that one can often simply call the job, with no parameters, and the orb will take care of the rest.

• Job 3: orb-tools/publish-dev

- orb-tools/publish-dev:
    orb-name: circleci/azure-cli
    context: orb-publishing
    requires:
      - orb-tools/pack

We’re almost at the end of our first workflow. This job simply publishes a dev version of our orb, which has already had its YAML linted and validated in previous jobs. Again, sensible defaults are provided so that one may only need to provide a value for the orb-name parameter. We’ve attached a context to this job, because publishing an orb requires a CircleCI API token, but if you’ve stored yours as a project environment variable, then a context might not even be necessary.

• Job 4: orb-tools/trigger-integration-workflow

- orb-tools/trigger-integration-workflow:
    name: trigger-integration-dev
    context: orb-publishing
    ssh-fingerprints: 23:d1:63:44:ad:e7:1a:b0:45:5e:1e:e4:49:ea:63:4e
    requires:
      - orb-tools/publish-dev
    filters:
      branches:
        ignore: master

- orb-tools/trigger-integration-workflow:
    name: trigger-integration-master
    context: orb-publishing
    ssh-fingerprints: 23:d1:63:44:ad:e7:1a:b0:45:5e:1e:e4:49:ea:63:4e
    tag: master
    requires:
      - orb-tools/publish-dev
    filters:
      branches:
        only: master

Here, things get a bit more complex. We’ve finished all our basic orb validation and published a dev version of our orb. Now we want to trigger our second workflow, which will only run if a git tag is pushed back to our orb repository. That is exactly what this orb-tools job does! This is why the job requires SSH fingerprints as a parameter. The job creates a git tag and pushes it, via SSH, back to our orb repository.

Our documentation walks through the process of creating a key and adding it to CircleCI, but in short: generate a passwordless OpenSSL (not OpenSSH) key pair, store the public portion as a read/write key in your GitHub or Bitbucket repository, and store the private portion in CircleCI. This will enable your CircleCI jobs to make changes to your repo e.g., pushing a git tag.

This type of approach is necessary, because internally, our build system processes configuration files, including orbs, at runtime. Thus, without a new webhook event (like a git tag), we would not be able to test the new dev version of our orb without manually pushing a second commit. The git tag causes our system to re-process our config.yml file, pulling in the just-published dev release of our orb.

Crucially, because the git tag is attached to the same commit as the initial one we pushed from our local machine, this new CircleCI build is seen as part of the same set of VCS status checks. That is, for a particular pull request, both sets of workflow jobs will be attached to the same commit, making it easy for developers to evaluate a given code change.

You’ll notice that we call the trigger-integration-workflow job twice. That’s because our integration-testing workflow uses regular expression filtering to control which of its jobs will run, depending on the content of the git tag. With this, we can accomplish the following:

1. Run only our integration-testing jobs whenever a commit is pushed to a non-master branch; whereas a commit to a master branch will trigger integration-testing jobs plus the possibility of a production deployment, if testing jobs succeed.
2. Dynamically publish either a patch, minor, or major release, depending on which parts of our orb source code have been modified in a given commit. (Example: a new command or job will trigger a possible major release; a new executor, or a change to the orb’s overall description field, will trigger only a minor release.) This feature is turned on by default, but can be disabled (set the use-git-diff parameter to false) if you’d like more manual control over when various types of orb releases are published.

The second trigger-integration-workflow job, running only on commits to master, essentially appends the string “master” to our git tag, causing our full second workflow, with possible production deployment jobs, to run.

Workflow 2:

integration-tests_prod-deploy

Orb integration tests will always be a bit more free-form than what you saw with our first workflow, which can essentially be drag-and-dropped into almost any orb repo’s CircleCI configuration file. My basic approach, as someone who manages a large (and growing!) number of orbs, is to focus on usage testing, covering as many edge cases as is reasonably possible.

In other words, take the dev version of our orb that we just published in the previous workflow, and run all of its commands and jobs. We do this to make sure they don’t fail, and to make sure they do what we expect. If a command or job can be used in multiple, fairly distinct ways, we run it multiple times so that we cover them all. If a command or job involves interacting with a third-party service, in this case, Microsoft Azure, we set up a test environment so we can have confidence that the orb will work for other users.

You might even take the approach we’ve taken with this Azure CLI orb and test the same commands or jobs (if the jobs are configured to take custom executors) with different runtime environments (below, you’ll see we test the orb’s commands using a Go-based Docker image, a Python-based Docker image, and Microsoft’s own first-party Azure Docker image).

Since the specifics of this example may not apply to other orbs, I’ll include the full YAML for the second workflow, but touch only briefly on a few points before moving on to discuss the production deployment job, which is also abstracted out into our Orb Tools orb and is designed to be usable by any orb developer looking to implement a similar development process.

YAML anchor filters used in our integration-testing jobs:

integration-dev_filters: &integration-dev_filters
  branches:
    ignore: /.*/
  tags:
    only: /integration-.*/

integration-master_filters: &integration-master_filters
  branches:
    ignore: /.*/
  tags:
    only: /master-.*/

prod-deploy_requires: &prod-deploy_requires
  [test-orb-python_master, test-orb-azure_master, test-orb-golang_master]

The full integration-testing and deployment workflow:

integration-tests_prod-deploy:
  jobs:
    # triggered by non-master branch commits
    - test-orb-python:
        name: test-orb-python_dev
        context: orb-publishing
        filters: *integration-dev_filters

    - test-orb-azure-docker:
        name: test-orb-azure_dev
        context: orb-publishing
        filters: *integration-dev_filters

    - test-orb-golang:
        name: test-orb-golang_dev
        context: orb-publishing
        filters: *integration-dev_filters

    # triggered by master branch commits
    - test-orb-python:
        name: test-orb-python_master
        context: orb-publishing
        filters: *integration-master_filters

    - test-orb-azure-docker:
        name: test-orb-azure_master
        context: orb-publishing
        filters: *integration-master_filters

    - test-orb-golang:
        name: test-orb-golang_master
        context: orb-publishing
        filters: *integration-master_filters

    # patch, minor, or major publishing
    - orb-tools/dev-promote-prod:
        name: dev-promote-patch
        orb-name: circleci/azure-cli
        context: orb-publishing
        requires: *prod-deploy_requires
        filters:
          branches:
            ignore: /.*/
          tags:
            only: /master-patch.*/

    - orb-tools/dev-promote-prod:
        name: dev-promote-minor
        orb-name: circleci/azure-cli
        release: minor
        context: orb-publishing
        requires: *prod-deploy_requires
        filters:
          branches:
            ignore: /.*/
          tags:
            only: /master-minor.*/

    - orb-tools/dev-promote-prod:
        name: dev-promote-major
        orb-name: circleci/azure-cli
        release: major
        context: orb-publishing
        requires: *prod-deploy_requires
        filters:
          branches:
            ignore: /.*/
          tags:
            only: /master-major.*/

The contents of the various test-orb jobs are defined earlier in this orb’s config.yml file. I encourage you to take a look, if you’re curious. For now, I only want to note our use of YAML anchors, mostly to simplify what would otherwise be some not-very-DRY sets of branch/tag filters. We’re using these filters to ensure that the git tag we pushed in the first workflow will correctly control whether or not our integration-testing jobs will lead to a production deployment.

Most importantly, I want to go over the orb-tools/dev-promote-prod job which we call three separate times.

• Last job in workflow 2: orb-tools/dev-promote-prod
  

Why are we calling this job three times?

This job is designed to promote a dev orb to a semantic/production orb version. As such, it takes a parameter determining whether the promoted version will be a patch (0.0.x), a minor (0.x.0) release, or a major (x.0.0) release. Because of how we are using git tags in the trigger-integration-workflow job to determine which type of release might ultimately be published, we need to call this job three times, with different regular expression filtering for git tags each time.

Other than that, this job is super straightforward! It’s designed to require as little boilerplate as possible. All that’s needed is an orb name and a CircleCI API token. If no release type is provided, it will default to patch.

Wrapping up

And that’s the end of our automated orb development workflow(s)! We just followed a single commit to a repository containing source code for an orb, through linting, basic orb validation, the publication of a dev orb release, triggering of an entire second integration-testing workflow via git tags, and dynamically interpolated production deployment of a patch, minor, or major orb version.

A few concluding notes:

This process makes use of a @dev:alpha tag for your orb. Whereas production orb versions are immutable, dev versions can be overwritten. This is what enables our config to be processed once with an older @dev:alpha version of this orb, and then re-processed after we’ve pushed a git tag with a newer @dev:alpha version. Thus, you will need to manually publish an initial @dev:alpha orb release to “bootstrap” this process, so that your config can be processed.

You may also need to occasionally re-publish @dev:alpha versions before pushing a given commit, if that commit has changed your orb source to the extent that jobs and commands are now being called that did not exist in the previous @dev:alpha version. Your CircleCI jobs will not run if their config cannot be processed, and calling orb commands that do not yet exist will cause a config processing error.

Since most of the testing jobs described here reside in our Orb Tools orb, we’ve added a full example of this process to that repository’s README.

Thanks for reading!

---

Read more:

• Tips for writing your first orb
• How to make an easy and valuable open source contribution with CircleCI orbs