On This Page
This guide introduces some basic concepts to help you understand how CircleCI manages your CI/CD pipelines.
In CircleCI, concurrency refers to utilizing multiple containers to run multiple jobs at the same time. To keep the system stable for all CircleCI customers, we implement different soft concurrency limits on each of the resource classes for different executors. If you are experiencing queueing on your jobs, it is possible you are hitting these limits. Customers on annual plans can request an increase to those limits at no extra charge.
CircleCI believes in configuration as code. Your entire CI/CD process is orchestrated through a single file called
config.yml file is located in a folder called
.circleci at the root of your project. CircleCI uses the YAML syntax for config. See the Writing YAML document for a basic introduction.
├── .circleci │ ├── config.yml ├── README └── all-other-project-files-and-folders
.circleci/config.yml is a powerful YAML file that defines the entire pipeline for your project. For a full overview of the various keys used, see the Configuration Reference page.
Your CircleCI configuration can be adapted to fit many different needs of your project. The following terms, sorted in order of granularity and dependence, describe the components of most common CircleCI projects:
- Pipeline: Represents the entirety of your configuration. (not available for server v2.x)
- Workflows: Responsible for orchestrating multiple jobs.
- Jobs: Responsible for running a series of steps that perform commands.
- Steps: Run commands (such as installing dependencies or running tests) and shell scripts to do the work required for your project.
The following illustration uses an example Java application to show the various config elements:
Contexts provide a mechanism for securing and sharing environment variables across projects. The environment variables are defined as name/value pairs and are injected at runtime. After a context has been created, you can use the
context key in the workflows section of a project
config.yml file to give any job(s) access to the environment variables associated with the context.
See Using Contexts for more information.
Data persistence allows you to move data between jobs and speed up your build. There are three main methods for persisting data in CircleCI: caches, workspaces, and artifacts.
A cache stores a file or directory of files such as dependencies or source code in object storage. To speed up the build, each job may contain special steps for caching dependencies from previous jobs.
Workspaces are a workflow-aware storage mechanism. A workspace stores data unique to the job, which may be needed in downstream jobs. Each workflow has a temporary workspace associated with it. The workspace can be used to pass along unique data built during a job to other jobs in the same workflow.
Artifacts persist data after a workflow is completed and may be used for longer-term storage of the outputs of your build process.
Note the following distinctions between artifacts, workspaces, and caches:
|Artifacts||Months||Preserve long-term artifacts.||Available in the Artifacts tab of the Job page under the |
|Workspaces||Duration of workflow||Attach the workspace in a downstream container with the ||The |
|Caches||Months||Store non-vital data that may help the job run faster, for example npm or Gem packages.||The |
See Persisting Data in Workflows: When to Use Caching, Artifacts, and Workspaces guide for additional conceptual information about using workspaces, caching, and artifacts.
Docker Layer Caching
Docker Layer Caching (DLC) caches the individual layers of Docker images built during your CircleCI jobs. Any unchanged layers are used on subsequent runs, rather than rebuilding the image each time.
config.yml snippet below, the
build_elixir job builds an image using the
ubuntu-2004:202104-01 Dockerfile. Adding
docker_layer_caching: true below the
machine executor key ensures CircleCI saves each Docker image layer as the Elixir image is built.
version: 2.1 jobs: build_elixir: machine: image: ubuntu-2004:202104-01 docker_layer_caching: true steps: - checkout - run: name: build Elixir image command: docker build -t circleci/elixir:example .
On subsequent commits, if the Dockerfile has not changed, DLC pulls each Docker image layer from cache during the
build Elixir image step and the image builds significantly faster.
See Docker Layer Caching for more information.
Instead of manually creating your configuration for each CircleCI project, you can generate this configuration dynamically, based on specific pipeline parameters or file paths. This is especially helpful where your team is working on a monorepo (or a single repository). Dynamic configuration allows you to trigger builds from specific parts of your project, rather than rebuilding everything each time.
See Dynamic Configuration for more information.
Each separate job defined within your config runs in a unique Execution environment. We call them executors. An executor can be a Docker container or a virtual machine running Linux, Windows, or macOS.
You can define an image for each executor. An image is a packaged system that includes instructions for creating a running container or virtual machine. CircleCI provides a range of images for use with the Docker executor, we call these convenience images. For more information, see the Pre-Built CircleCI Docker Images guide.
The primary container is defined by the first image listed in
.circleci/config.yml file. This is where commands are executed. The Docker executor spins up a container with a Docker image. The machine executor spins up a complete Ubuntu virtual machine image. See Introduction to Execution Environments document for a comparison table and considerations. Further images can be added to spin up secondary/service containers.
For added security when using the Docker executor and running Docker commands, the
setup_remote_docker key can be used to spin up another Docker container in which to run these commands. For more information see the Running Docker Commands guide.
Note: macOS is not available on installations of CircleCI server v2.x.
An image is a packaged system that includes instructions for creating a running container. The primary container is defined by the first image listed in a
.circleci/config.yml file. This is where commands are executed for jobs, using the Docker or machine executor.
The Docker executor spins up a container with a Docker image. CircleCI maintains convenience images for popular languages on Docker Hub.
The machine executor spins up a complete Ubuntu virtual machine image, giving you full access to OS resources and complete control over the job environment. For more information, see the Using machine page.
See the Introduction to Execution Environments document for a comparison.
version: 2.1 jobs: build1: # job name docker: # Specifies the primary container image, # see circleci.com/docs/circleci-images/ for # the list of pre-built CircleCI images on dockerhub. - image: cimg/base:2022.04-20.04 auth: username: mydockerhub-user password: $DOCKERHUB_PASSWORD # context / project UI env-var reference - image: postgres:14.2 # Specifies the database image auth: username: mydockerhub-user password: $DOCKERHUB_PASSWORD # context / project UI env-var reference # for the secondary or service container run in a common # network where ports exposed on the primary container are # available on localhost. environment: # Specifies the POSTGRES_USER authentication # environment variable, see circleci.com/docs/env-vars/ # for instructions about using environment variables. POSTGRES_USER: user ... build2: machine: # Specifies a machine image that uses # an Ubuntu version 20.04 image. # The image uses the current tag, which always points to the most recent # supported release. If stability and determinism are crucial for your CI # pipeline, use a release date tag with your image, e.g. ubuntu-2004:202201-02 image: ubuntu-2004:current ... build3: macos: # Specifies a macOS virtual machine with Xcode version 12.5.1 xcode: "12.5.1" ...
Jobs are the building blocks of your config. Jobs are collections of steps, which run commands/scripts as required. Each job must declare an executor that is either
docker you must specify an image to use for the primary container. For
macos you must specify an Xcode version. For
windows you must use the Windows orb.
Orbs are reusable snippets of code that help automate repeated processes, accelerate project setup, and make it easy to integrate with third-party tools. See Using Orbs for details on how to use orbs in your config and an introduction to orb design. Visit the Orbs Registry to search for orbs to help simplify your config.
The illustration in the Configuration section showing an example Java configuration could be simplified using orbs. The following illustration demonstrates a simplified configuration with the Maven orb. Here, the orb sets up a default executor that can execute steps with Maven and run a common job (
The more tests your project involves, the longer it takes for them to complete on a single machine. With parallelism, you can spread your tests across a specified number of separate executors.
Test suites are conventionally defined at the job level in your
.circleci/config.yml file. The
parallelism key specifies how many independent executors will be set up to run the steps of a job.
To run a job’s steps in parallel, set the
parallelism key to a value greater than 1.
# ~/.circleci/config.yml version: 2.1 jobs: test: docker: - image: cimg/<language>:<version TAG> auth: username: mydockerhub-user password: $DOCKERHUB_PASSWORD # context / project UI env-var reference parallelism: 4
See Running Tests in Parallel for more information.
A CircleCI pipeline is the full set of processes you run when you trigger work on your projects. Pipelines encompass your workflows, which in turn coordinate your jobs. This is all defined in your project configuration file. Pipelines are not available on CircleCI server v2.x.
Pipelines represent methods for interacting with your configuration:
- Trigger a pipeline via the API with the trigger a pipeline endpoint.
- Use pipeline parameters to trigger conditional workflows.
version 2.1config, which provides access to:
Note, it is important to carefully consider the impact of enabling the auto-cancel feature, for example, if you have configured automated deployment jobs on non-default branches.
A CircleCI project shares the name of the associated code repository in your Version Control System (VCS). Select Projects in the CircleCI web app sidebar to enter the projects dashboard. From here you can set up and follow the projects you have access to.
On the Projects Dashboard, you can either:
- Set Up any project that you are the owner of in your VCS.
- Follow any project in your organization to gain access to its pipelines and to subscribe to email notifications for the project’s status.
A resource class is a configuration option that allows you to control available compute resources (CPU and RAM) for your jobs. When you specify an execution environment for a job, a default resource class value for the environment will be set unless you define the resource class in your configuration. It is best practice to define the resource class, as opposed to relying on a default.
The example below shows how to define a resource class in the Docker execution environment.
jobs: build: docker: - image: cimg/node:current auth: username: mydockerhub-user password: $DOCKERHUB_PASSWORD # context / project UI env-var reference resource_class: large
Examples for all execution environments are available on the following pages:
- Using the Docker execution environment
- Using the LinuxVM execution environment
- Using the macOS execution environment
- Using the Windows execution environment
- Using the GPU execution environment
- Using the Arm execution environment
Pricing and plans information for the various resource classes can be found on the Resource Classes product page.
resource_class key is also used to configure a self-hosted runner instance.
Steps are usually a collection of the executable commands required to complete your job. For example, the
checkout step (which is a built-in step available across all CircleCI projects) checks out the source code for a job over SSH. The
run step allows you to run custom commands, such as executing the command
make test, using a non-login shell by default. Commands can also be defined outside the job declaration, making them reusable across your config.
#... jobs: build: docker: - image: <image-name-tag> auth: username: mydockerhub-user password: $DOCKERHUB_PASSWORD # context / project UI env-var reference steps: - checkout # Special step to checkout your source code - run: # Run step to execute commands, see # circleci.com/docs/configuration-reference/#run name: Running tests command: make test # executable command run in # non-login shell with /bin/bash -eo pipefail option # by default. #...
Here are the user types relating to CircleCI projects. Many of them have permissions inherited from VCS accounts
- The Organization Administrator is a permission level inherited from your VCS:
- GitHub: Owner and following at least one project building on CircleCI.
- Bitbucket: Admin and following at least one project building on CircleCI.
- The Project Administrator is the user who adds a GitHub or Bitbucket repository to CircleCI as a Project.
- A User is an individual user within an organization, inherited from your VCS.
- A CircleCI user is anyone who can log in to the CircleCI platform with a username and password. Users must be added to a GitHub or Bitbucket org to view or follow associated CircleCI projects. Users may not view project data that is stored in environment variables.
Workflows orchestrate jobs. A workflow defines a list of jobs and their run order. It is possible to run jobs concurrently, sequentially, on a schedule, or with a manual gate using an approval job.
The following config example shows a workflow called
build_and_test in which the job
build1 runs and then jobs
build3 run concurrently:
Your First Green Build guides you step-by-step through setting up a working pipeline.
Help make this document better
This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.
- Suggest an edit to this page (please read the contributing guidefirst).
- To report a problem in the documentation, or to submit feedback and comments, please open an issue on GitHub.
- CircleCI is always seeking ways to improve your experience with our platform. If you would like to share feedback, please join our research community.
Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.
You can also visit our support site to find support articles, community forums, and training resources.
CircleCI Documentation by CircleCI is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.