Designing a Package Manager

Less than three months ago, we launched CircleCI orbs, our package management ecosystem for CircleCI configuration. In that time we have seen over 300 orbs created in just the first 10 weeks and more than 200 organizations using orbs across over 1,000 projects in tens of thousands of builds. We are also featuring more than 20 official partner orbs through our new Technology Partner Program. We’re incredibly grateful for the huge and positive response.

In this piece, I want to take a look back and share the key design decisions that went into creating and launching CircleCI orbs. If you’re using orbs today and want to understand how to get the most from them, or if you’re thinking about designing your own package management system, this post is for you.

What are CircleCI orbs?

Orbs are shareable packages of CircleCI configuration. Like most other code-packaging systems, orbs provide a common set of abstract structural elements, making reuse of tested, encapsulated build components practical across your organization as well as the broader ecosystem of users. Orbs have a domain-specific language (DSL) defined in YAML syntax that expresses the semantics of our build configuration. This makes it straightforward to compose idiomatic interfaces for encapsulated commands, jobs, and executors in your build configuration.

Orbs also provide package management infrastructure. This makes it straightforward to publish clearly versioned orbs and then import them for use in projects.

What problems were orbs designed to solve?

Our job is to help teams animate and accelerate their software delivery. One of the key ingredients to getting the most from CircleCI is our build configuration, a YAML-based semantics to describe the workflows, jobs, and environments of CI pipelines. When we started building orbs we wanted to solve three key problems with regards to the build configuration:

1. Better DRY support
The configuration in CircleCI 2.0 is designed to be highly transparent, expressive, and deterministic. However, those traits can also make our base semantics verbose and repetitive. Without nice shortcuts and sugaring, configurations can have lots of boilerplate code, hindering project setup and making the config unwieldy as it grows in sophistication and scope.
2. Code reuse across projects
Many customers want better ways to share configuration across projects. CircleCI configuration was not previously easy to share across teams, creating unnecessary redundancy and slowing teams down. Our customers who have large teams want ways to easily disseminate and enforce common patterns and policies and to help their teams improve through shared practices.
3. Easier paths to common configuration
Developers try to avoid reinventing things or re-solving solved problems. When bootstrapping code, they want to use off-the-shelf, honed encapsulations whenever it’s practical and reliable. In CircleCI, starting from a blank page can be daunting for first-time users, and we wanted to make it easier to get from nothing to something useful.

Key design decisions (and why we made them)

Designing a new package management system was, as we expected, not a straightforward process. In building orbs we needed to balance approachability, expressiveness, portability, and determinism. We reviewed other package management systems as well as various configuration templating approaches and quickly converged on several of the key elements that we wanted to incorporate.

Because orbs are more focused in their scope than most general purpose package managers, we had the freedom to be more opinionated in our design decisions. Along the way, we did internal and external previews of various bits of the design, allowing us to get quick feedback before cementing anything in code.

Before getting into the design of orbs themselves, let’s start with some context on the underlying CircleCI configuration model we were building upon.

Existing Decisions about CircleCI Config

When we started the design process for CircleCI orbs we knew we’d be sticking with the existing design principles of CircleCI configuration:

1. Configuration in code: Always prefer allowing the code itself to contain the configuration of what needs to be done, unless it’s absolutely necessary for a setting or other input to exist outside of code (eg: project settings intended to be applied to all branches automatically or secrets injection and security, intended to live outside of the code base).

2. Deterministic: Every build should behave the same way given the same inputs. This prevents things from changing on you that you are unable to control, while alleviating CircleCI from having to make and then carry assumptions about how you want to run your builds.

3. Config as data, not programs: YAML is used because it provides a way to author a data structure with a reasonable balance between syntactic weight and expressiveness. Having configuration be data rather than a programming language allows easy and reliable tooling for doing config syntax and schema validation, various static analysis and data transformations that occur during the processing of your builds, and automated documentation with first-class metadata. For instance, the configuration of Jenkins plugins is not, itself, executable code. Orbs, on the other hand, can be reasoned about as configuration composition rather than as executable software in a particular environment. While this does not solve all dependency and security concerns, it creates cleaner boundaries between how you want to configure builds from what happens inside the runtime environment(s) during execution.

4. Declarative of build behaviors: The semantics of CircleCI configurations revolve around the core domain model of build execution inside our platform. The core structure is driven by workflows that invoke and coordinate jobs that express a set of steps to run and the execution environment in which to run them. Our configuration code is intrinsically meta to your build process, in that we provide the superstructure in which you express your own runtime commands and process orchestration. We want to let you focus on what your code is going to do inside the runtime and get out of your way as much as possible. Using a declarative syntax allows to give you an expressive set of primitives without making you learn much, if anything, about how our internal systems work or learn some new programming language.

The existing design of CircleCI configuration became our key constraint when setting out to build orbs. Beyond those constraints, there are three key areas of orb design (in no particular order):

  • The design of the orb registry
  • The development lifecycle of orbs
  • The semantics of our domain-specific language (DSL)

I’ll go into each of these areas, and the decisions we made, below.

Designing the Orb Registry

Our original designs proposed starting with “local” orbs that lived inside your .circleci folder that would be pulled in at runtime. Orbs inside .circleci were deemed unacceptably risky because of the way we interact with GitHub. “Do not treat git as a package manager” was the clear warning from our internal review of the situation. This created the need to have our own registry, which definitely added substantial scope (on the order of 60-90 elapsed days) and created a clear design boundary going forward.

1. All orbs live in a namespace: All orbs live in exactly one namespace. There is no “empty” namespace, nor are there reserved special defaults like _ for CircleCI or “official” orbs. We decided that we didn’t want orbs we author to be considered the default set or have special significance in our namespacing scheme. We are likely to introduce a Certified Orbs program or similar in the future, but that will offer first-class metadata on the orb, explicitly defining inclusion rather than relying on implications from a special namespace.

2. Semantic versioning and locking: CircleCI wants to allow orb authors to dynamically add features and fixes to orbs. At the same time, we want to prevent things from changing out from under orb users: fixing the orbs that users employ so that users’ configurations remain static unless that user specifies otherwise. CircleCI chose to enforce a strict format for semantic versioning on all published revisions. When importing we allow an orb user to lock a specific revision or to assume the risk of breakage and to use the latest version (volatile).

3. “Volatile” over “latest”: We chose to use the word “volatile” over “latest” because we want to encourage use of specific versions for greater determinism. We want someone using volatile to understand they are doing something that comes with risk of breakage. We decided that “latest” sounds fresh and new, whereas “volatile” better expresses what’s happening when you are willing to take upstream dependencies that could arbitrarily mutate without warning. Also worth noting is that we don’t default to “volatile” or any version – you must specify at least a partial semver or “volatile” when invoking orbs.

4. Certified vs 3rd party orbs: Certified means that CircleCI treats them as part of our platform. Anyone can use them without additional opt-in if they are using 2.1 configuration or higher. For now, only the circleci namespace is Certified because we have written those orbs ourselves. All others are considered 3rd party.

5. Open vs private orbs: All orbs are open, meaning they are world-readable. Anyone can use them and see their source. We did not want to introduce black boxes into the runtime environment of your code and secrets, so it’s important that if you’re going to run orbs in your jobs you should be able to see what they do. We are likely to add some way to have private orbs in the future.

Orb Development Decisions

At the heart of the orbs system is how developers who author orbs do their work. The priority here was to balance ease of iteration and starting small with the durability and immutability in mind, allowing users of orbs the ability to incorporate them into their builds. Following are the key characteristics of the orb authoring experience.

1. Revisions are immutable: To prevent mysterious changes cropping up in the builds of those who use orbs, we do not allow changes to an orb revision once it has gone live. To publish a newer version of code you need to publish a new revision.

2. Dev vs production orbs: Once we decided to enforce strict, immutable versioning of production orbs, we wanted a nice way to let you work on developing orbs without polluting your semver progression. Thus, orbs can be published either as “dev:foo” or as a semantically versioned production orb. Dev orbs are mutable by anyone on your team and expire 90 days after their last publish date. Production orbs are immutable and durable, so that your build configuration is immune to unexpected changes in orbs you use.

3. Register-time dependency resolution: If an orb (my-orb) imports other orbs, we will resolve and lock those dependencies at the time that my-orb is added to the registry. For instance, let’s say that you publish version 1.2.0 of my-orb, and it contains this orb invocation: foo-orb: somenamespace/some-orb:volatile – At the time you register 1.2.0 of my-orb the latest version of foo-orb will be flattened into my-orb version 1.2.0. If a new version of foo-orb is published it won’t get incorporated into my-orb until you publish a new version. We recommend selecting the fully qualified version of the orbs that you import to ensure deterministic behavior.

4. Transparency: If you can execute an orb you can see the source of that orb. This is so you are never executing a black box inside your jobs, where your code and secrets are present. Orbs are less like your core software, and more like a piece of configuration you would otherwise copy and paste. Orbs are part of your build process, so you should be able to see what they do if you can execute them.

DSL Design Decisions

Inside orbs, the syntax and semantics of our YAML-based Domain Specific Language (DSL) needed to be enhanced to allow for parameterization and reuse. Following are some of the key decisions we made on this front.

1. Orbs package specific config elements rather than provide a generalized templating solution. Looking around at other tools, we saw a few instances of configuration that used generic templating for sharing snippets of code across projects. We decided early to focus orbs on using the semantics of our configuration DSL and allow invoking particular elements, so that using parameters could be done with specific scoping, and to avoid unexpected conflicts. This approach also makes the configuration easier to read and reason about.

2. Baked-in/first-class metadata. CircleCI allows a description key as part of the structure of orbs, so it’s easier to generate documentation or read through the code without having extra-structural comments.

3. Orbs are a subset of config.yml. You may notice that the configuration format in your config.yml file is very similar to that of orbs. Early on, it became clear that config.yml is, effectively, a special kind of orb – you could think of it as the “root orb” for running your build on CircleCI.

What’s Next?

We’re excited to see so many teams publishing and using orbs in just a few weeks. Throughout 2019 we hope to add a few new features to orbs and continue to build out our own orbs and work with our partners to make integrating with their systems even easier. Below are several resources to learn more or participate in helping us shape the future of build configuration.

To learn more about using or authoring orbs see the documentation:

To see which orbs are available to use in your configuration see the Orb Registry:

To make feature requests or vote up existing requests see our ideas board:

For general discussion about orbs see our Discuss board:

Related blog posts: