Build private CircleCI orbs on any organization

Using CircleCI’s orbs is a great way to share CI/CD configuration across projects. Public orbs work well for wide adoption, but private orbs have been helpful for organizations needing to share common internal configuration in a secure, non-public way. Private orbs work only within the organization that publishes them.

We recently opened up private orbs access to all CircleCI customers, including those on the Free plan. This new access opens up a number of new possibilities for your pipelines.

In this article you will learn about private orbs and how to use them for your own organization. We will be using the sample orb I built, which contains a command and a job that lets you trigger CircleCI pipelines in other projects using cURL. You can find the source code of the project in this repository.

Prerequisites

This article assumes:

• Working knowledge of CircleCI
• Understanding of how pipelines work
• CircleCI CLI installed and configured locally on your machine
• You are using either a project organization or your personal organization on GitHub

Use cases for private orbs

Like orbs in general, private orbs are useful for sharing steps and tasks that are common to multiple projects.

Many organizations have their own frameworks and internal tools used across many teams. With private orbs they can share those tools easily, without opening it up to the wider world. Some use cases for private orbs include deployment to your proprietary Kubernetes clusters or other infrastructure, dealing with custom hardware, all the way to logging or common admin tasks.

Initial setup

We will use the CircleCI CLI for this task. With the CLI you will create a new orb, and register it with CircleCI so it can be referenced by other CircleCI pipelines. The CLI allows you to release new versions of your orb.

The whole orb creation process is documented in the CircleCI docs. This article will serve as a quick recap of it and explanation of the steps involved.

Create an empty git repository and note its path. I created mine under my zmarkan-demos organization, and named it api-requester-orb-sample, so the path is: git@github.com:zmarkan-demos/api-requester-orb-sample.git.

Next, create the actual orb with the CLI’s circleci orb init command, passing in a local path where you want the orb source to be located. This directory must be new; do not use one that already exists. In my case, this was:

circleci orb init ~/github/api-requester-orb-sample --private

The main difference between public and private orbs is the --private flag at the end.

The CLI asks you to either follow the guided setup, which is what I did, or to do it yourself. I recommend using the automated process is recommended as it will set up everything in the directory with an empty orb template, as well as link it to your repository, create a namespace for your orb, and create a CircleCI project for the new orb, just make sure to copy the git URL from the repository you created earlier.

Finally you need to push the local code to the remote repository -

cd ~/github/api-requester-orb-sample
git push origin master

Note: All orbs exist in a namespace within your organization. This is the same for any runner agents you have registered already.

You can also review your new private orbs in your organization settings, where they are clearly labeled as private under the type section.

Developing an orb

The orb development process depends on whether you opt for the automated guided process or follow your own. As with the setup section, the CircleCI docs cover both scenarios.

If you have followed the quick setup route, then the CLI will have generated the whole project for you from template. It has samples for a command, job, executor, and a test for it.

You can remove any part of the orb that you do not need. In my case I removed the scripts and executors, and kept the commands and jobs. Orbs are just a collection of CircleCI configuration scripts that get resolved and bundled when you run a pipeline.

We created 2 scripts:

• A commands/trigger-pipeline.yml command
• A jobs/trigger-circleci-pipeline.yml job

The job wraps the command and executes in the base Docker executor.

Using the trigger-pipeline command

The trigger-pipeline command has 4 parameters, and a single step, as shown in this code sample:

description: >
  This command triggers CircleCI pipeline specified.
  The job execution environment must have cURL utility available

parameters:
  repo-name:
    type: string
    description: "Repo name to trigger pipeline for"

  branch-name:
    type: string
    default: main
    description: "Branch name to trigger"

  trigger-params-json:
    type: string
    default: "{}"
    description: Pipeline trigger parameters in JSON format

  circle-token-env-var:
    type: env_var_name
    default: CIRCLE_TOKEN
    description: Environment variable containing your CircleCI API Key. Default is $CIRCLECI_API_KEY

steps:
  - run:
      name: Run cURL request
      command: |
        curl --request POST \
            --url "https://circleci.com/api/v2/project/gh/${CIRCLE_PROJECT_USERNAME}/<< parameters.repo-name >>/pipeline" \
            --header "Circle-Token: ${<< parameters.circle-token-env-var >>}" \
            --header "content-type: application/json" \
            --data '{"branch":"<< parameters.branch-name >>","parameters":<< parameters.trigger-params-json >>}"'

Unlike a full CircleCI pipeline config, this one does not require a version stanza. With only a single command source, it doesn’t have jobs or workflows either. All 4 parameters are also specific to this command, unlike the pipeline parameters available in the full CircleCI config.

The curl command in the run step hits the CircleCI API to trigger another pipeline. It is described in detail in this blog post.

Using the trigger-circleci-pipeline job

Writing a job that consumes the command we just created is straightforward. Within the same orb, you can use that command as if it were a local command you had defined in the config. In our example in jobs/trigger-circleci-pipeline.yml it looks like this:

description: >
  Job that trigger CircleCI pipeline in another project with the payload specified

docker:
  - image: cimg/base:2021.12

parameters:
  repo-name:
    type: string
    description: "Repo name to trigger pipeline for"
  branch-name:
    type: string
    default: main
    description: "Branch name to trigger"

  trigger-params-json:
    type: string
    default: "{}"
    description: Pipeline trigger parameters in JSON format

  circle-token-env-var:
    type: env_var_name
    default: CIRCLE_TOKEN
    description: Environment variable containing your CircleCI API Key. Default is $CIRCLECI_API_KEY

steps:
  - trigger-pipeline:
      repo-name: << parameters.repo-name >>
      branch-name: << parameters.branch-name >>
      trigger-params-json: << parameters.trigger-params-json >>
      circle-token-env-var: << parameters.circle-token-env-var >>

This command has the parameters, and as with all other jobs in CircleCI we need the executor: docker in our case. The steps stanza contains a single step, invoking our trigger-pipeline command from before, with any parameters we set.

Testing and deploying the orb

To build, test, and deploy our orb, the orb template project uses CircleCI itself. Within the config in .circleci/config.yml most everything is pre-made for you. All you need to do is to write tests for it. By default, it contains a job named integration-test-1, which you can modify to use your own orb. Here is a snippet that invokes the trigger-pipeline command:

jobs:
  integration-test-1:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - api-requester-orb-sample/trigger-pipeline:
          repo-name: pinging-me-softly
          branch-name: main
          trigger-params-json: '{"image-name": "cimg/base:2021.12"}'

To publish a release version you need to:

• Change from the default alpha branch to either the master or main branch
• Include a [semver:patch|minor|major] message in the commit

Conclusion

In this article you have learned about building private orbs for CircleCI, which are now available to all CircleCI customers. Private orbs allow you to have shared CI/CD pipeline configuration across multiple projects. These private orbs can be used by any team member in your organization, without the risk of exposing your config to the public.

The process as a whole is almost identical for publishing public orbs. Just add the --private flag in the circleci orb init command.

To read more about orb development check out the docs pages on authoring orbs.

To read about testing and verifying bash scripts in your orbs using BATS and ShellCheck, check out this docs page on orb testing methodologies.

If you have any questions or suggestions about this article, or ideas for future articles and guides, reach out to me on Twitter - @zmarkan or email me.