An introduction to pipelines

A CI/CD pipeline is the most fundamental component of automated software development. While the term has been used to describe many different aspects of computer science, at CircleCI, and within much of the DevOps industry, we use “pipeline” to illustrate the broad applications of behaviors and processes involved in continuous integration (CI).

CI is a software development strategy that increases development speed while ensuring the quality of code deployed is not compromised. Using CI tools, developers continually commit code in small increments, sometimes multiple times a day, which is then automatically built and tested before it is merged with the shared repository. Modern software delivery pipelines can build, test, and deploy applications based on your business needs.

What is a CI/CD Pipeline?

A CI/CD pipeline is the full set of processes that run when you trigger work on your projects. Pipelines encompass your workflows, which coordinate your jobs, and this is all defined in your project configuration file.

Continuous integration (CI) vs continuous deployment (CD)

Continuous integration automates the building and testing of your software. Continuous deployment is an extension of this automation and allows for your software to be deployed after every code commit that passes your test suite. The most successful development teams deploy their software often. For more information, read our Continuous integration (CI) vs continuous deployment (CD) page.

Components of a CI/CD pipeline

Continuous integration automates the building and testing of your software. Continuous deployment is an extension of this automation and allows for your software to be deployed after every code commit that passes your test suite.

The most successful development teams deploy their software often. To deliver high-quality software efficiently means building, testing, and deploying code using CI/CD best practices.

Build stage

In the build stage, multiple development teams contribute code developed on their own machines into a shared repository. This sounds simple but quickly introduces complexities. Beyond version control, problems can arise including subtle differences in developer and production environments, tooling, and quality of code. The advantage of including the build process in your pipeline is that it automates developer contributions and provides tools to standardize software quality and environments.

Testing stage

All too often, development teams move directly to the deploy stage. This is a mistake because the testing stage is where the key benefits of CI/CD shine. Testing is a complex and repetitive process that your CI/CD pipeline helps automates for you.

There are several different types of testing all of which can be used together in an automated continuous integration pipeline. You can combine unit testing with integration testing to provide the most test coverage possible. Testing also contributes vital data about software performance that can immediately be integrated back into the code. The result of testing is high quality software with fewer and fewer bugs.

Deploy stage

The deploy stage is where you can orchestrate software releases to production or other environments. Your pipeline can be configured to deploy code on a schedule, roll out software to all customers or just a select group, and even roll back releases when there is a problem. You get to decide what the best strategy is for getting updated software to your customers. It can all be automated as part of your CI/CD pipeline.

Configuring a CI pipeline

The DNA of continuous integration is configuration. CI pipelines are the highest level of orchestration in a config file.

CI Workflows

Workflows allow you to run and troubleshoot jobs separately so you can see failed builds in real-time. If one job in a workflow fails, you can re-run that job alone instead of re-running the entire set.

An image of workflows from CircleCI's UI

CI Jobs

Jobs are collections of steps that are executed in a single unit, and workflows are the set of rules that define that set of jobs and their run order.

An image of jobs from CircleCI's UI

Configuration steps

Within jobs, steps are a list of single key-value pairs. The key indicates the type of step and the value can be either a configuration map or a string. When the step is a run, you can specify which command to execute as a string value.

An image of steps from CircleCI's UI

CI/CD features with CircleCI

Some of the features you’ll encounter when building code via pipelines on CircleCI are the use of multiple resource classes, parallel test splitting, orbs, matrix jobs, environment variables/contexts, and approvals.

  • Resource classes: CircleCI offers a number of resource classes that allow you to optimize CPU and RAM resources for each job.

  • Parallel test splitting: To reduce time, run tests in parallel by spreading them across multiple, separate containers.

  • Orbs: Orbs are reusable packages of YAML configuration that condense repeated pieces of config into single lines of code.

  • Matrix jobs: Matrix jobs allow you to run a parameterized job multiple times with different arguments.

  • Parameters: Pipeline variables, environment variables, and contexts are parameters that allow users to store and reuse data and to protect sensitive information.

  • Approvals: Workflows can be programmed to wait for manual approval of a job before moving on. Anyone with push access to your repository can approve the job to continue the workflow.

Start building with pipelines on CircleCI

If you’re new to continuous integration, understanding what pipelines do and are capable of will help you understand the true value of CI. To learn more, visit the CircleCI resources page. To get started, check out our docs.