Pipelines overview

Cloud

Server v4.x

Server v3.x

CircleCI pipelines are the highest-level unit of work, and they are described by a configuration file, for example, .circleci/config.yml. Triggers can be set up to automate when a pipeline will be triggered. Pipelines coordinate the execution of workflows and jobs to build, test, and deploy your code. Workflows can be conditional, based on filters, job requirements, and pipeline parameters.

Pipelines include your workflows, which coordinate your jobs. Pipelines have a fixed, linear lifecycle, and are associated with a specific actor. Pipelines trigger when a change is pushed to a project that has a CircleCI configuration file included, and can also be scheduled, triggered manually through the CircleCI app, or using the API.

Dashboard

When visiting your CircleCI dashboard, you are shown a list of recently-triggered pipelines for your organization.

Pipeline architecture

A pipeline is composed of workflows, which are composed of jobs. By navigating from a pipeline to a specific job, you can access your job output, test results, and artifacts through several tabs.

The output of each job can be opened in a new tab (in either raw or formatted styling) with a unique link, making it shareable between team members.

Pipelines and triggers

The information in this section on pipelines and triggers applies to accounts integrated with the CircleCI GitHub App or GitLab. There are two GitHub integration types, either OAuth or through the GitHub App. To find out which GitHub integration type you have refer to the GitHub App integration page.

Set up pipelines and triggers to define when and how pipelines are automatically triggered for a project. These triggers may be from external services (custom webhooks), or from commits or tags on a version control system.

Pipelines

Pipelines define units of work that can be run within your projects. Each pipeline defines a config source, which points to the location of a CircleCI configuration file. The config source can be set up to be used when triggering a pipeline. Pipelines can be associated with one or more triggers.

A project can have one or more pipelines. The default config source location is .circleci/config.yml. If you authenticated through the GitHub App, you can set this location to any path to a *.yml file.

When you create a project in CircleCI and connect it to a repository, a pipeline is set up for you automatically, and connected to a config source. The connected configuration file is either a file you commit yourself or one generated by CircleCI based on the code in your project.

Add or edit a pipeline

If you created a project in the CircleCI web app you will find a pipeline is already set up.

To add or edit a pipeline, follow these steps:

In the CircleCI web app, select Projects in the sidebar.
Select the ellipsis next to your project () and choose Project Settings.
Select Pipelines in the sidebar.
Select Add Pipeline, or, if you wish to edit an existing pipeline select the pencil icon on the right.

Complete the form fields and options:

Give your pipeline a descriptive name, for example Run tests and deploy.

Under Config Source select the platform that matches where your project is set up (for example, GitHub App).

For GitLab Self-managed, you can input any instance that you have previously set up in CircleCI. If you wish to set a different feature branch or repository from a self-managed instance as a new pipeline, you will first need to add a new connection via your Integrations. In either case, you will also need to enter your personal access token again to authorize this connection.

Authorize your connection if this is not already showing with a

Select your repository

Select the same repository for your pipeline and trigger (described in the following section). There is currently a bug that means that CircleCI will always look for the pipeline in the same repository from which the triggering event was emitted.

Select a fallback branch to use in case the triggering event does not have a triggering branch. This branch will be used if you trigger this pipeline via a custom webhook.
Enter the path to your configuration file relative to the root of your repository. For example, .circleci/config.yml. If you are integrated through the GitHub App, you can enter any path to a .yml file, and the file can have any name you choose.

Select Save.

Once you have set up a pipeline you need to set up a trigger to connect it to. This is described in the following section.

Triggers

If you created a project in the CircleCI web app you will find a trigger is already set up.

Triggers are rules to define that a pipeline should be run on a given event. A project can have one or more triggers, which are associated with events from a source of change. The most common type of trigger is a VCS event, for example a push to a branch on a GitHub repository, but any system that can emit outbound webhooks can also be set up as a trigger. A trigger can have only one pipeline but a pipeline can have multiple triggers.

A trigger kicks off a pipeline to run the workflows defined in your connected config source.

Add a trigger

To add a trigger for a pipeline, follow these steps:

When a trigger is created, CircleCI registers a webhook with GitLab. Push events from GitLab are sent to CircleCI. CircleCI then uses the event data to determine if a pipeline should run, and if so, which pipeline should be run.

In addition to a pipeline, each trigger includes the webhook URL, and in this scenario, a CircleCI-created GitLab token. The webhook URL and GitLab token are used to securely register the webhook within GitLab in order to receive push events from your GitLab repository. You can view webhooks for a project in GitLab at Settings Webhooks.

To add a trigger for a pipeline, follow these steps:

In the CircleCI web app select Projects in the sidebar
Find your project in the list, select the ellipsis () next to it and choose Project Settings.
Select Triggers in the sidebar.
Select Add Trigger.
Select the same location in the "Connect to" dropdown menu that you selected for your pipeline (for example, GitLab).
Select Next.

Complete the form fields and options:

Give your trigger a descriptive name.

Authorize your connection if this is not already showing with a (Not required for custom webhooks).

For GitLab self-managed you can enter the URL for an instance you have previously set up with CircleCI. You will need to enter the relevant personal access token again here to authorize the connection.

Select your repository from the dropdown menu. This should match the repository your pipeline is connected to (not required for custom webhooks).
Choose your pipeline from the "Choose config to run" menu.
(Optional) You can configure trigger filters.

Select Save

When setting up a trigger you will see a modal titled Webhook URL requesting that you set up a webhook in GitLab. You do not need to take action. The webhook is set up automatically by CircleCI. This is a known issue and will be fixed.

To verify your trigger is set up correctly, trigger an event from your repository.

Trigger filters allow you to determine when a trigger should initiate a build based on the parameters provided by Gitlab’s webhook. CircleCI provides some common options, for example, only build on merge requests, but you can also build your own rules using the custom filter option. For example, a custom filter would allow you to only build on a specific branch or user.

VS Code extension

If you use Visual Studio Code, you can also monitor and interact with your pipelines directly from VS Code with the official CircleCI extension. The extension allows you to customize which projects and pipelines you want to follow, as well as view job logs and test results, download artifacts, approve, re-run, and debug jobs with SSH, and get notified when your workflows fail or need approval.