Start Building for Free
CircleCI.comAcademyBlogCommunitySupport

Concurrency

2 months ago2 min read
Cloud
Server 3
Server 2
On This Page

This page summarizes concurrency, and suggests ways to use concurrency to your advantage.

In CircleCI, concurrency refers to utilizing multiple containers to run multiple jobs at the same time. This is different from CircleCI parallelism, which is test-splitting across multiple containers. If you would like information on parallelism, visit the Running Tests in Parallel page.

Every plan with CircleCI includes the ability to run jobs concurrently. However, the number of jobs you can run concurrently differs between plans. Please refer to the pricing page for up-to-date information on the number of concurrent job runs included in your plan.

Concurrency

Within CircleCI, concurrent jobs can be classified as one of two situations:

  • Multiple running jobs configured in a single workflow

  • Multiple workflows running by multiple members within the organization

This page discusses the first point, and will provide examples of how to configure concurrent jobs in a single workflow. Configuring concurrency in a workflow allows multiple jobs to be run at the same time, in a single workflow, through the use of multiple containers.

To keep the system stable for all CircleCI customers, we implement different soft concurrency limits on each of the resource classes for different executors.

If jobs are queued, it is possible you are hitting these limits. Hitting your limit may depend on a few factors, such as how many people in your organization are running jobs at once, or if you have overlapping workflows.

Customers on annual plans can request an increase to those limits at no extra charge by contacting CircleCI support.

Concurrency in workflows

To run a set of concurrent jobs, you will need to add a workflows section to your existing .circleci/config.yml file.

The simple example below shows the default workflow orchestration with two concurrent jobs. The workflows key needs to have a unique name. In this example, the unique name is build_and_test. The jobs key is nested under the uniquely named workflow, and contains the list of job names. Since the jobs have no dependencies, they will run concurrently.

version: 2.1

jobs:
  build:
    docker:
      - image: cimg/<language>:<version TAG>
        auth:
          username: mydockerhub-user
          password: $DOCKERHUB_PASSWORD  # context / project UI env-var reference
    steps:
      - checkout
      - run: <command>
  test:
    docker:
      - image: cimg/<language>:<version TAG>
        auth:
          username: mydockerhub-user
          password: $DOCKERHUB_PASSWORD  # context / project UI env-var reference
    steps:
      - checkout
      - run: <command>
workflows:
  build_and_test:
    jobs:
      - build
      - test

Please note, if you are still using Server 2 and version: 2, your workflows section will need to specify the version again, as in the below snippet:

...
workflows:
  version: 2
  build_and_test:
    jobs:
      - build
      - test
...

Fan-out/fan-in workflow example

A more complex example of using concurrent workflows is running a common build job, fanning-out to run a set of acceptance test jobs concurrently, and then fanning-in to run a common deploy job. This flow is illustrated below.

Fan-out and Fan-in Workflow

The .circleci/config.yml snippet below is an example of a workflows section configured for fan-out/fan-in job execution. In this example, as soon as the build job finishes successfully, all four acceptance test jobs start. The deploy job must wait for all four acceptance test jobs to complete successfully before it starts.

...
workflows:
  build_accept_deploy:
    jobs:
      - build
      - acceptance_test_1:
          requires:
            - build
      - acceptance_test_2:
          requires:
            - build
      - acceptance_test_3:
          requires:
            - build
      - acceptance_test_4:
          requires:
            - build
      - deploy:
          requires:
            - acceptance_test_1
            - acceptance_test_2
            - acceptance_test_3
            - acceptance_test_4
...

Please note, if you are still using CircleCI Server 2 with version: 2, you will need to specify the version in the workflows section, as mentioned in the Concurrency in workflows section.


Help make this document better

This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.

Need support?

Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.

You can also visit our support site to find support articles, community forums, and training resources.