Pipelines and triggers overview

Cloud

Server v4+

CircleCI pipelines are the highest-level unit of work in your CI/CD process. Defined by a configuration file (typically .circleci/config.yml), pipelines orchestrate the execution of workflows which run jobs to build, test, and deploy your code.

Pipelines can be triggered in the following ways:

Automatically: Git events, scheduled events, custom webhooks
Manually: CircleCI app or API

Pipelines provide a flexible framework for creating conditional workflows based on filters, job requirements, and pipeline parameters, allowing you to customize your CI/CD process to fit your project’s needs.

Pipelines dashboard

Your pipelines dashboard displays recently-triggered pipelines for your organization. Use the various filters to find the pipelines you need. Select workflow and job names to view workflow maps and individual job steps. Use the quick controls on the right of the screen to rerun and cancel workflows.

From the pipelines dashboard you can find the pipeline ID. The pipeline ID is a unique ID for a pipeline that has been triggered. You can use the pipeline ID to access pipeline information via the API.

Screenshot showing the location on the pipeline ID

Pipelines and triggers

The pipeline and trigger features available to you depend on how your code is integrated and where it is stored. For a feature availability overview, see the VCS integration overview page.

Using CircleCI server? The Pipeline and Trigger pages under Project Settings and pipeline and trigger add/edit functionality is not available.

Set up pipelines to define the various CI/CD "units" that make up your project.

Set up triggers to define when and how each of your pipelines should be run. Triggers may be from external services (custom webhooks), from commits or tags on a version control system, or set up to run on a schedule.

Pipelines

A pipeline defines a unit of work that can be run within your projects. Each pipeline is defined by two key characteristics:

A config source, which is a configuration file stored in a repository. The default config source location is .circleci/config.yml within the same repository as your code.

For GitHub App and Bitbucket Data Center pipelines you can choose any file path (with extension *.yml) in any repository.
A checkout source, which determines which repository will be checked out when running a checkout step.

This model allows your CircleCI config to be stored in a different repository from your code. This is currently supported for GitHub App and Bitbucket Data Center pipelines.

A project can have one or more pipelines, and each pipeline can be associated with zero or more triggers.

When you create a project in CircleCI and follow the in-app project creation flow to connect it to a repository, a first pipeline is set up for you automatically. The config file used for your first pipeline is either a file you commit yourself or one generated by CircleCI based on the code in your project. You can identify the type (GitHub App, GitHub OAuth) of a pipeline using the badges in the Config Source column on the Project Settings Pipelines page.

Each pipeline has a Definition ID, which is a UUID for the pipeline, which can be used to refer to a specific pipeline when triggering it via the API.

Add or edit a pipeline

If you created a project in the CircleCI web app you will find a pipeline is already set up.

GitHub and Bitbucket OAuth pipelines are created automatically when setting up a project and they cannot be edited.

The steps and options described in this section are for setting up a GitHub App pipeline. If you are setting up a pipeline for a different version control system the options you see may vary but the process is mostly the same.

To add or edit a pipeline, follow these steps:

In the CircleCI web app, select Projects in the sidebar.
Select the ellipsis next to your project () and choose Project Settings.
Select Pipelines in the sidebar.
Select Add Pipeline, or, if you wish to edit an existing pipeline select the pencil icon on the right.

Complete the form fields and options:

Give your pipeline a descriptive name, for example Run tests and deploy.

Under Integration select the platform that matches where your project is set up (for example, GitHub App).

If you integrate with CircleCI’s GitHub OAuth App you can select GitHub App here to install the App into your org and access improved security and new features. This is a one-time step, which can be done by an organization admin or a user who has admin access to at least one repository in the organization. Once installed, further GitHub App pipelines can be setup without this installation step.

For GitLab self-managed, you can input any instance that you have previously set up in CircleCI. If you wish to set a different feature branch or repository from a self-managed instance as a new pipeline, you will first need to add a new connection via your Integrations. In either case, you will also need to enter your personal access token again to authorize this connection.

Authorize your connection if this is not already showing with a check mark ().
Define where the config file for this pipeline is stored:
- Select a repository from the dropdown.
- Enter the path to the config file.
Under Build Assets, select a Checkout Source for this pipeline. This is the repository that will be checked out when a checkout step is run. If your config is stored in the same repository as your code, pick the same repository you selected for the Config Source.

Select Save.

Once you have set up a pipeline you need to set up a trigger to connect it to. Setting up a trigger is described in the following section.

Triggers

If you created a project in the CircleCI web app you will find a trigger is already set up.

GitHub and Bitbucket OAuth triggers are created automatically when setting up a project and they cannot be edited. The only trigger you can create and edit for an OAuth pipeline is a scheduled pipeline trigger.

Triggers are configurable rules that determine when to automatically run a pipeline based on specific events and conditions. A project can have one or more triggers, which are associated with events from a source of change. The most common type of trigger is a VCS event, for example a push to a branch on a GitHub repository, but any system that can emit outbound webhooks can also be set up as a trigger. A trigger can have only one pipeline but a pipeline can have multiple triggers.

A trigger kicks off a pipeline to run the workflows defined in your connected config source.

Add a trigger

To add a trigger for a pipeline, follow these steps:

Custom webhooks are available for projects integrated via the GitHub App. Custom webhooks are not available on CircleCI server.

Setting up a custom webhook trigger allows you to trigger pipelines from external services. Any service that can emit a webhook or make a cURL request can be set up in this way to trigger a pipeline. For more information, see the Triggers overview page.

Custom webhook triggers can only be associated with GitHub App pipelines. You can check if you have a GitHub App pipeline by going to Project Settings Pipelines and checking if any of your pipelines have a GitHub App badge in the Config Source column. If needed, create a new pipeline by selecting Add pipeline.

To add a trigger for a custom webhook, follow these steps:

In the CircleCI web app select Projects in the sidebar.
Find your project in the list, select the ellipsis (), select Project Settings, and select Pipelines in the sidebar.
If there is already a GitHub App pipeline set up and it uses the desired YAML configuration file, proceed to the Triggers step below (step 5). If not, select Set up pipeline. If there is a Connect button, select it and follow the steps to install the CircleCI GitHub App into your GitHub organization (this step is compatible with orgs that integrate with CircleCI’s GitHub OAuth app).
Name the new pipeline and select the repository where the CircleCI YAML configuration file is stored. Enter the file path (including the .circleci directory). The file path can be different than .circleci/config.yml (for example, .circleci/webhook.yml). Save the pipeline.
Select Triggers in the sidebar.
Select Add Trigger.
Select Custom Webhook from the dropdown menu.
Select Next.
Complete the form fields and options:
- Enter a descriptive name for the trigger. For example, if you are setting up a custom webhook to run pipelines on events from Datadog, enter "Datadog" here.
- (Optional) Add a description.
- In the field "Pipeline to run", at the bottom of the page, select the pipeline that you created in the step above.
- Enter the branch to use to fetch your config file when a Custom Webhook is received.
- Enter the branch to use to check out your code when using the checkout step in config. If your config is stored in the same repository as your code, then your Config branch and your Checkout branch should be the same.
Select Save.
You will see a webhook endpoint URL and secret. You can use these to set up your webhook trigger from your external source. Copy the Webhook URL and use it in your trigger source appended with the secret.

The secret will not be shown again so be sure to copy the URL before clicking Done.

To verify your trigger is set up correctly, trigger an event from your third party system.

When a trigger is created, CircleCI registers a webhook with GitLab. Push events from GitLab are sent to CircleCI. CircleCI then uses the event data to determine if a pipeline should run, and if so, which pipeline should be run.

In addition to a pipeline, each trigger includes the webhook URL, and in this scenario, a CircleCI-created GitLab token. The webhook URL and GitLab token are used to securely register the webhook within GitLab in order to receive push events from your GitLab repository. You can view webhooks for a project in GitLab at Settings Webhooks.

To add a trigger for a pipeline, follow these steps:

In the CircleCI web app select Projects in the sidebar
Find your project in the list, select the ellipsis () next to it and choose Project Settings.
Select Triggers in the sidebar.

If you do not see Triggers in the sidebar, your project is integrated using either GitHub OAuth or Bitbucket Cloud and these steps do not apply.
Select Add Trigger.
Select the same location in the "Connect to" dropdown menu that you selected for your pipeline (for example, GitLab).
Select Next.

Complete the form fields and options:

Give your trigger a descriptive name.

Authorize your connection if this is not already showing with a (Not required for custom webhooks).

For GitLab self-managed you can enter the URL for an instance you have previously set up with CircleCI. You will need to enter the relevant personal access token again here to authorize the connection.

Select your repository from the dropdown menu. This should match the repository your pipeline is connected to (not required for custom webhooks).
Choose your pipeline from the "Choose config to run" menu.
(Optional) You can configure trigger filters to configure the event types that will trigger the pipeline. For more information, see the GitLab trigger options page.

Select Save

When setting up a trigger you will see a modal titled Webhook URL requesting that you set up a webhook in GitLab. You do not need to take action. The webhook is set up automatically by CircleCI. This is a known issue and will be fixed.

To verify your trigger is set up correctly, trigger an event from your repository.

Pipeline parameters

Pipeline parameters are declared using the parameters key at the top level of a .circleci/config.yml file. Pipeline parameters can be referenced by value and used as a configuration variable under the scope pipeline.parameters.

The example below shows a configuration with two pipeline parameters (image-tag and workingdir) defined at the top of the configuration, and then subsequently referenced in the build job:

version: 2.1

parameters:
  image-tag:
    type: string
    default: "current"
  workingdir:
    type: string
    default: "~/main"

jobs:
  build:
    docker:
      - image: cimg/node:<< pipeline.parameters.image-tag >>
    environment:
      IMAGETAG: << pipeline.parameters.image-tag >>
    working_directory: << pipeline.parameters.workingdir >>
    steps:
      - run: echo "Image tag used was ${IMAGETAG}"
      - run: echo "$(pwd) == << pipeline.parameters.workingdir >>"

A pipeline can be triggered with specific parameter values using the API v2 or via the CircleCI web app.

See the Pipeline values and parameters page for more information.

VS Code extension

If you use Visual Studio Code, you can also monitor and interact with your pipelines directly from VS Code with the official CircleCI extension. The extension allows you to customize which projects and pipelines you want to follow, as well as view job logs and test results, download artifacts, approve, re-run, and debug jobs with SSH, and get notified when your workflows fail or need approval.

Screenshot showing the detailed view of a failed test

The CircleCI VS Code extension is available to download on the VS Code marketplace.

Troubleshooting

Config could not be located error

If you see the following error message, check the steps below to remediate the issue.

config file .circleci/sample-filename.yml could not be located on branch sample-branch-name in repository sample-repo-name

Ensure that there is a CircleCI configuration file in the repository on the branch that uses the filename specified in the error message. If there is not one present, add a CircleCI configuration file.
If there is a config file present:
1. Navigate to Project Settings Pipelines in the CircleCI web app for the project where you are seeing this error message.
2. Select the pencil icon for each pipeline listed and ensure that the "Config File Path" field matches the filepath of the config file that is in your repository. If you changed the name of the config file in your repository, the reference to that filepath must also be changed in the Project Settings Pipelines section for any pipeline that uses that configuration file.

Why is my scheduled pipeline not running?

If your scheduled pipeline is not running, verify the following things:

Is the actor who is set for the scheduled pipelines still part of the organization? You can find this setting is under Attribution in the Triggers section of the web app.
Is the branch set for the schedule deleted?
Is your VCS organization using SAML protection? SAML tokens expire often, which can cause requests to fail.

Why are my jobs not running when I push commits?

In the CircleCI application, check the individual job and workflow views for error messages. More often than not, the error is because of formatting errors in your .circleci/config.yml file.

See the YAML Introduction page for more details.

After checking your .circleci/config.yml for formatting errors, search for your issue in the CircleCI support center.

Why is my job queued?

A job might end up being queued because of a concurrency limit being imposed due your organization’s plan. If your jobs are queuing often, you can consider upgrading your plan.

Why are my jobs queuing even though I am on the Performance Plan?

In order to keep the system stable for all CircleCI customers, we implement different soft concurrency limits on each of the Resource classes. If you are experiencing queuing on your jobs, it is possible you are hitting these limits. Contact CircleCI support to request raises on these limits.

Why can I not find my project on the Projects dashboard?

If you are not seeing a project you would like to build, and it is not currently building on CircleCI, check your org in the top left corner of the CircleCI application. For instance, if the top left shows your user my-user, only projects belonging to my-user will be available under Projects. If you want to build the project your-org/project, you must switch your organization on the application’s organization switcher menu to your-org.

How do Docker image names work? Where do they come from?

CircleCI currently supports pulling (and pushing with Docker Engine) Docker images from Docker Hub. For official images, you can pull by simply specifying the name of the image and a tag:

golang:1.7.1-jessie
redis:3.0.7-jessie

For public images on Docker Hub, you can pull the image by prefixing the account or team username:

my-user/couchdb:1.6.1

What is the best practice for specifying image versions?

It is best practice not to use the latest tag for specifying image versions. It is also best practice to use a specific version and tag, for example cimg/ruby:3.0.4-browsers, to pin down the image and prevent upstream changes to your containers when the underlying base distribution changes. For example, specifying only cimg/ruby:3.0.4 could result in unexpected changes from browsers to node. For more context, refer to Docker image best practices, and CircleCI image best practices.

How can I set the timezone in Docker images?

You can set the timezone in Docker images with the TZ environment variable. A sample .circleci/config.yml with a defined TZ variable would look like the following:

version: 2.1
jobs:
  build:
    docker:
      - image: your/primary-image:version-tag
        auth:
          username: mydockerhub-user
          password: $DOCKERHUB_PASSWORD  # context / project UI env-var reference
      - image: mysql:5.7
        auth:
          username: mydockerhub-user
          password: $DOCKERHUB_PASSWORD  # context / project UI env-var reference
        environment:
           TZ: "America/Los_Angeles"
    working_directory: ~/your-dir
    environment:
      TZ: "America/Los_Angeles"

In this example, the timezone is set for both the primary image and an additional mySQL image.

A full list of available timezone options is available on Wikipedia.

FAQs

Is it possible to split the `.circleci/config.yml` into different files?

If your code is stored in GitHub you can split your configuration into multiple files using the CircleCI GitHub App integration. For steps, see the Set up multiple configuration files how-to guide.

To find out which GitHub integration type you have, check your Organization slug at Organization settings Overview. OAuth authenticated orgs are structured as follows:

github/<your-org-name> or gh/<your-org-name>
bitbucket/<your-org-name> or bb/<your-org-name>

An organization slug for a GitHub App integration is in the following format:

circleci/<UID>

If your organization is currently integrated with GitHub only through OAuth, an admin can take the one-time action of installing the CircleCI GitHub App to enable this functionality. For more information, see the GitHub integration page.

While splitting configuration files is only available for GitHub App integrations, CircleCI does support dynamic configuration, which allows you to create configuration files based on specific pipeline values or paths, and continue one pipeline to another. See the Dynamic configuration page for more information.

Can I trigger forked PRs using pipelines?

The "build forked PRs" settings is not currently supported for GitLab or GitHub App projects. To find out if you authorized through the GitHub OAuth app or the CircleCI GitHub App, see the GitHub App integration page.

You can trigger pipelines to build PRs from forked repositories with CircleCI API v2. However, by default, CircleCI will not build a PR from a forked repository. If you would like to turn this feature on, navigate to Project Settings Advanced in the web app. If you would like more information, you can view this support article.

Next steps

Find out more about triggering pipelines in the Triggers Overview.

Suggest an edit to this page

Make a contribution

Learn how to contribute

Still need help?

Ask the CircleCI community

Join the research community

Visit our Support site