An introduction to pipelines

A CI/CD pipeline is the most fundamental component of automated software development. While the term has been used to describe many different aspects of computer science, at CircleCI, and within much of the DevOps industry, we use “pipeline” to illustrate the broad applications of behaviors and processes involved in continuous integration (CI).

CI is a software development strategy that increases development speed while ensuring the quality of code deployed is not compromised. Using CI tools, developers continually commit code in small increments, sometimes multiple times a day, which is then automatically built and tested before it is merged with the shared repository. Modern software delivery pipelines can build, test, and deploy applications based on your business needs.

What is a CI/CD Pipeline?

A CI/CD pipeline is the full set of processes that run when you trigger work on your projects. Pipelines encompass your workflows, which coordinate your jobs, and this is all defined in your project configuration file.

Continuous integration (CI) vs continuous deployment (CD)

Continuous integration automates the building and testing of your software. Continuous deployment is an extension of this automation and allows for your software to be deployed after every code commit that passes your test suite. The most successful development teams deploy their software often. For more information, read our Continuous integration (CI) vs continuous deployment (CD) page.

Configuring a CI pipeline

The DNA of continuous integration is configuration. CI pipelines are the highest level of orchestration in a config file.

CI Workflows

Workflows allow you to run and troubleshoot jobs separately so you can see failed builds in real-time. If one job in a workflow fails, you can re-run that job alone instead of re-running the entire set.

An image of workflows from CircleCI's UI

CI Jobs

Jobs are collections of steps that are executed in a single unit, and workflows are the set of rules that define that set of jobs and their run order.

An image of jobs from CircleCI's UI

Configuration steps

Within jobs, steps are a list of single key-value pairs. The key indicates the type of step and the value can be either a configuration map or a string. When the step is a run, you can specify which command to execute as a string value.

An image of steps from CircleCI's UI

CI/CD features with CircleCI

Some of the features you’ll encounter when building code via pipelines on CircleCI are the use of multiple resource classes, parallel test splitting, orbs, matrix jobs, environment variables/contexts, and approvals.

  • Resource classes: CircleCI offers a number of resource classes that allow you to optimize CPU and RAM resources for each job.

  • Parallel test splitting: To reduce time, run tests in parallel by spreading them across multiple, separate containers.

  • Orbs: Orbs are reusable packages of YAML configuration that condense repeated pieces of config into single lines of code.

  • Matrix jobs: Matrix jobs allow you to run a parameterized job multiple times with different arguments.

  • Parameters: Pipeline variables, environment variables, and contexts are parameters that allow users to store and reuse data and to protect sensitive information.

  • Approvals: Workflows can be programmed to wait for manual approval of a job before moving on. Anyone with push access to your repository can approve the job to continue the workflow.

Start building with pipelines on CircleCI

If you’re new to continuous integration, understanding what pipelines do and are capable of will help you understand the true value of CI. To learn more, visit the CircleCI resources page. To get started, check out our docs.