Pipelines overview

Cloud

Server v4.x

Server v3.x

CircleCI pipelines are the highest-level unit of work, and they are described by a configuration file, for example, .circleci/config.yml. Triggers can be set up to automate when a pipeline will be triggered. Pipelines coordinate the execution of workflows and jobs to build, test, and deploy your code. Workflows can be conditional, based on filters, job requirements, and pipeline parameters.

Pipelines include your workflows, which coordinate your jobs. Pipelines have a fixed, linear lifecycle, and are associated with a specific actor. Pipelines trigger when a change is pushed to a project that has a CircleCI configuration file included, and can also be scheduled, triggered manually through the CircleCI app, or using the API.

Dashboard

When visiting your CircleCI dashboard, you are shown a list of recently-triggered pipelines for your organization.

Pipeline architecture

A pipeline is composed of workflows, which are composed of jobs. By navigating from a pipeline to a specific job, you can access your job output, test results, and artifacts through several tabs.

The output of each job can be opened in a new tab (in either raw or formatted styling) with a unique link, making it shareable between team members.

Pipelines and triggers

Set up pipelines and triggers to define when and how pipelines are automatically triggered for a project. These triggers may be from external services (custom webhooks), or from commits or tags on a version control system.

Pipelines

Pipelines define units of work that can be run within your projects. Each pipeline is defined by two key characteristics:

A configuration file, which is stored in a repository referred to as config source, at a path of your choosing. The default config source location is .circleci/config.yml, but any file path with extension *.yml can be used.
A checkout source, which determines which repository will be checked out when running a checkout step. This model allows your CircleCI config to be stored in a different repository from your code.

A project can have one or more pipelines, and each pipeline can be associated with one or more triggers.

When you create a project in CircleCI and connect it to a repository, a first pipeline is set up for you automatically. The connected configuration file is either a file you commit yourself or one generated by CircleCI based on the code in your project.

Add or edit a pipeline

If you created a project in the CircleCI web app you will find a pipeline is already set up.

To add or edit a pipeline, follow these steps:

In the CircleCI web app, select Projects in the sidebar.
Select the ellipsis next to your project () and choose Project Settings.
Select Pipelines in the sidebar.
Select Add Pipeline, or, if you wish to edit an existing pipeline select the pencil icon on the right.

Complete the form fields and options:

Give your pipeline a descriptive name, for example Run tests and deploy.

Under Integration select the platform that matches where your project is set up (for example, GitHub App). If you integrate with CircleCI’s GitHub OAuth App) you can select GitHub App here to install the App into your org and access improved security and new features.

For GitLab Self-managed, you can input any instance that you have previously set up in CircleCI. If you wish to set a different feature branch or repository from a self-managed instance as a new pipeline, you will first need to add a new connection via your Integrations. In either case, you will also need to enter your personal access token again to authorize this connection.

Authorize your connection if this is not already showing with a
Define where the config file for this pipeline is stored:
- Select a repository from the dropdown.
- Enter the path to the config file.
Under Build Assets, select a checkout Source for this pipeline. This is the repository that will be checked out when a checkout step is run. If your config is stored in the same repository as your code, pick the same repository you selected for the field Config Source.

Select Save.

Once you have set up a pipeline you need to set up a trigger to connect it to. This is described in the following section.

Triggers

If you created a project in the CircleCI web app you will find a trigger is already set up.

Triggers are rules to define that a pipeline should be run on a given event. A project can have one or more triggers, which are associated with events from a source of change. The most common type of trigger is a VCS event, for example a push to a branch on a GitHub repository, but any system that can emit outbound webhooks can also be set up as a trigger. A trigger can have only one pipeline but a pipeline can have multiple triggers.

A trigger kicks off a pipeline to run the workflows defined in your connected config source.

Add a trigger

To add a trigger for a pipeline, follow these steps:

Setting up a custom webhook trigger allows you to trigger pipelines from external services. Any service that can emit a webhook or make a cURL request can be set up in this way to trigger a pipeline. For more information, see the Triggers overview page.

To add a trigger for a custom webhook, follow these steps:

In the CircleCI web app select Projects in the sidebar.
Find your project in the list, select the ellipsis (), select Project Settings, and select Pipelines in the sidebar.
If there is already a GitHub App pipeline set up and it uses the desired YAML configuration file, proceed to the Triggers step below (step 5). If not, select Set up pipeline. If there is a Connect button, select it and follow the steps to install the CircleCI GitHub App into your GitHub organization (this step is compatible with orgs that integrate with CircleCI’s GitHub OAuth app).
Name the new pipeline and select the repository where the CircleCI YAML configuration file is stored. Enter the file path (including the .circleci directory). The file path can be different than .circleci/config.yml (for example, .circleci/webhook.yml). Save the pipeline.
Select Triggers in the sidebar.
Select Add Trigger.
Select Custom Webhook from the dropdown menu.
Select Next.
Complete the form fields and options:
- Enter a descriptive name for the trigger. For example, if you are setting up a custom webhook to run pipelines on events from Datadog, enter "Datadog" here.
- (Optional) Add a description.
- In the field "Pipeline to run", at the bottom of the page, select the pipeline that you created in the step above.
- Enter the branch to use to fetch your config file when a Custom Webhook is received.
- Enter the branch to use to check out your code when using the checkout step in config. If your config is stored in the same repository as your code, then your Config branch and your Checkout branch should be the same.
Select Save.
You will see a webhook endpoint URL and secret. You can use these to set up your webhook trigger from your external source. Copy the Webhook URL and use it in your trigger source appended with the secret.

The secret will not be shown again so be sure to copy the URL before clicking Done.

To verify your trigger is set up correctly, trigger an event from your third party system.

When a trigger is created, CircleCI registers a webhook with GitLab. Push events from GitLab are sent to CircleCI. CircleCI then uses the event data to determine if a pipeline should run, and if so, which pipeline should be run.

In addition to a pipeline, each trigger includes the webhook URL, and in this scenario, a CircleCI-created GitLab token. The webhook URL and GitLab token are used to securely register the webhook within GitLab in order to receive push events from your GitLab repository. You can view webhooks for a project in GitLab at Settings Webhooks.

To add a trigger for a pipeline, follow these steps:

In the CircleCI web app select Projects in the sidebar
Find your project in the list, select the ellipsis () next to it and choose Project Settings.
Select Triggers in the sidebar.

If you do not see Triggers in the sidebar, your project is integrated using either GitHub OAuth or Bitbucket Cloud and these steps do not apply.
Select Add Trigger.
Select the same location in the "Connect to" dropdown menu that you selected for your pipeline (for example, GitLab).
Select Next.

Complete the form fields and options:

Give your trigger a descriptive name.

Authorize your connection if this is not already showing with a (Not required for custom webhooks).

For GitLab self-managed you can enter the URL for an instance you have previously set up with CircleCI. You will need to enter the relevant personal access token again here to authorize the connection.

Select your repository from the dropdown menu. This should match the repository your pipeline is connected to (not required for custom webhooks).
Choose your pipeline from the "Choose config to run" menu.
(Optional) You can configure trigger filters.

Select Save

When setting up a trigger you will see a modal titled Webhook URL requesting that you set up a webhook in GitLab. You do not need to take action. The webhook is set up automatically by CircleCI. This is a known issue and will be fixed.

To verify your trigger is set up correctly, trigger an event from your repository.

Trigger filters allow you to determine when a trigger should initiate a build based on the parameters provided by Gitlab’s webhook. CircleCI provides some common options, for example, only build on merge requests, but you can also build your own rules using the custom filter option. For example, a custom filter would allow you to only build on a specific branch or user.

VS Code extension

If you use Visual Studio Code, you can also monitor and interact with your pipelines directly from VS Code with the official CircleCI extension. The extension allows you to customize which projects and pipelines you want to follow, as well as view job logs and test results, download artifacts, approve, re-run, and debug jobs with SSH, and get notified when your workflows fail or need approval.

Screenshot showing the detailed view of a failed test

The CircleCI VS Code extension is available to download on the VS Code marketplace.

Troubleshooting

Config could not be located error

If you see the following error message, check the steps below to remediate the issue.

config file .circleci/sample-filename.yml could not be located on branch sample-branch-name in repository sample-repo-name

Ensure that there is a CircleCI configuration file in the repository on the branch that uses the filename specified in the error message. If there is not one present, add a CircleCI configuration file.
If there is a config file present:
1. Navigate to Project Settings Pipelines in the CircleCI web app for the project where you are seeing this error message.
2. Select the pencil icon for each pipeline listed and ensure that the "Config File Path" field matches the filepath of the config file that is in your repository. If you changed the name of the config file in your repository, the reference to that filepath must also be changed in the Project Settings Pipelines section for any pipeline that uses that configuration file.

Why is my scheduled pipeline not running?

If your scheduled pipeline is not running, verify the following things:

Is the actor who is set for the scheduled pipelines still part of the organization? You can find this setting is under Attribution in the Triggers section of the web app.
Is the branch set for the schedule deleted?
Is your VCS organization using SAML protection? SAML tokens expire often, which can cause requests to fail.

Why are my jobs not running when I push commits?

In the CircleCI application, check the individual job and workflow views for error messages. More often than not, the error is because of formatting errors in your .circleci/config.yml file.

See the YAML Introduction page for more details.

After checking your .circleci/config.yml for formatting errors, search for your issue in the CircleCI support center.

Why is my job queued?

A job might end up being queued because of a concurrency limit being imposed due your organization’s plan. If your jobs are queuing often, you can consider upgrading your plan.

Why are my jobs queuing even though I am on the Performance Plan?

In order to keep the system stable for all CircleCI customers, we implement different soft concurrency limits on each of the Resource classes. If you are experiencing queuing on your jobs, it is possible you are hitting these limits. Contact CircleCI support to request raises on these limits.

Why can I not find my project on the Projects dashboard?

If you are not seeing a project you would like to build, and it is not currently building on CircleCI, check your org in the top left corner of the CircleCI application. For instance, if the top left shows your user my-user, only projects belonging to my-user will be available under Projects. If you want to build the project your-org/project, you must switch your organization on the application’s organization switcher menu to your-org.

How do Docker image names work? Where do they come from?

CircleCI currently supports pulling (and pushing with Docker Engine) Docker images from Docker Hub. For official images, you can pull by simply specifying the name of the image and a tag:

golang:1.7.1-jessie
redis:3.0.7-jessie

For public images on Docker Hub, you can pull the image by prefixing the account or team username:

my-user/couchdb:1.6.1

What is the best practice for specifying image versions?

It is best practice not to use the latest tag for specifying image versions. It is also best practice to use a specific version and tag, for example cimg/ruby:3.0.4-browsers, to pin down the image and prevent upstream changes to your containers when the underlying base distribution changes. For example, specifying only cimg/ruby:3.0.4 could result in unexpected changes from browsers to node. For more context, refer to Docker image best practices, and CircleCI image best practices.

How can I set the timezone in Docker images?

You can set the timezone in Docker images with the TZ environment variable. A sample .circleci/config.yml with a defined TZ variable would look like the following:

version: 2.1
jobs:
  build:
    docker:
      - image: your/primary-image:version-tag
        auth:
          username: mydockerhub-user
          password: $DOCKERHUB_PASSWORD  # context / project UI env-var reference
      - image: mysql:5.7
        auth:
          username: mydockerhub-user
          password: $DOCKERHUB_PASSWORD  # context / project UI env-var reference
        environment:
           TZ: "America/Los_Angeles"
    working_directory: ~/your-dir
    environment:
      TZ: "America/Los_Angeles"

In this example, the timezone is set for both the primary image and an additional mySQL image.

A full list of available timezone options is available on Wikipedia.

Next steps

Find out more about triggering pipelines in the Triggers Overview.

Suggest an edit to this page

Make a contribution

Learn how to contribute

Still need help?

Ask the CircleCI community

Join the research community

Visit our Support site