Documentation structure for LLMs (llms.txt)

Serial group recipes

Cloud Server

Server version: 4.9 and later

These recipes show how to use serial groups to run jobs one at a time. For how serial groups work, the queueing rules, and how to release a group from the web app, see Controlling Serial Execution Across Your Organization.

Each example is a complete configuration file that you can adapt. Replace the ./scripts/ commands with your own.

Run one deployment at a time across your organization

Goal

You may want to stop deployments to a shared environment from overlapping when several projects deploy to it.

Steps

  1. Add the serial-group key to the deploy job in every project that deploys to the environment. Use the same literal string in each project:

    # Use the same serial-group string in every project that deploys to staging.
    # Only one deploy job with this value runs at a time across your organization.
    version: 2.1
    
    jobs:
      test:
        docker:
          - image: cimg/base:current
        steps:
          - checkout
          - run: ./scripts/test.sh
      deploy:
        docker:
          - image: cimg/base:current
        steps:
          - checkout
          - run: ./scripts/deploy.sh staging
    
    workflows:
      main-workflow:
        jobs:
          - test
          - deploy:
              serial-group: deploy/staging
              requires:
                - test
  2. Trigger a run in two of the projects at the same time. The second deploy job shows a queued status until the first deploy job finishes.

Notes

  • A serial group applies across your organization. Two jobs in different projects are in the same group when their serial-group values are identical.

  • Do not add << pipeline.project.slug >> to the value. It gives each project its own group, so deployments from different projects no longer queue behind each other. To serialize within one project only, see Run one deployment at a time for each project.

  • The next job in the queue starts when a job finishes, whatever the job’s final status.

  • A job that requires the deploy job runs only if the deploy job succeeded. Releasing the lock does not change this.

  • A job waits in the queue for up to five hours. After that, CircleCI cancels it.

  • If a newer pipeline in the same project is waiting or running in the group, a job from an older pipeline is skipped. To roll back to an earlier version, rerun a workflow from the earlier pipeline when no job in the group is waiting or running.

Run one deployment at a time for each project

Goal

You may want pipelines in one project to deploy one at a time, for example when several merges to main happen close together, without making other projects wait.

Steps

  • Add the serial-group key to the deploy job and include the project slug in the value:

    # The project slug makes the group name unique to this project.
    # Pipelines in this project queue behind each other. Other projects are not affected.
    version: 2.1
    
    jobs:
      test:
        docker:
          - image: cimg/base:current
        steps:
          - checkout
          - run: ./scripts/test.sh
      deploy:
        docker:
          - image: cimg/base:current
        steps:
          - checkout
          - run: ./scripts/deploy.sh
    
    workflows:
      main-workflow:
        jobs:
          - test
          - deploy:
              serial-group: << pipeline.project.slug >>/deploy
              requires:
                - test

Notes

  • The << pipeline.project.slug >> pipeline value is different in every project, so each project has its own group.

  • The / in the project slug is a permitted character in a serial-group value.

Run one deployment at a time for each environment

Goal

You may want staging and production deployments to queue separately, so a slow production deployment does not hold up staging.

Steps

  • Add a pipeline parameter for the environment and use it in the serial-group value. Each value of the parameter has its own group:

    # Each value of deploy-target gets its own group.
    # A staging deploy never waits for a production deploy, and the reverse.
    version: 2.1
    
    parameters:
      deploy-target:
        type: enum
        enum: ["staging", "production"]
        default: "staging"
    
    jobs:
      test:
        docker:
          - image: cimg/base:current
        steps:
          - checkout
          - run: ./scripts/test.sh
      deploy:
        parameters:
          target:
            type: string
        docker:
          - image: cimg/base:current
        steps:
          - checkout
          - run: ./scripts/deploy.sh << parameters.target >>
    
    workflows:
      main-workflow:
        jobs:
          - test
          - deploy:
              target: << pipeline.parameters.deploy-target >>
              serial-group: deploy/<< pipeline.parameters.deploy-target >>
              requires:
                - test

Notes

  • Use a parameter with a fixed set of values, such as an enum, so the group name is always valid. A serial-group value must be 512 characters or fewer and can contain only alphanumeric characters and ., -, _, and /.

  • To use a specific value, set the parameter when you trigger a run. See Trigger Options.

Run database migrations one at a time across services

Goal

You may want to stop two services from migrating the same database at the same time.

Steps

  • In each service that shares the database, add the serial-group key to the migration job. Use the same literal string in every service. Jobs that depend on the migration, such as the deploy job, stay outside the group:

    # Every service that migrates the shared database uses this same serial-group string.
    # Only one migrate job runs at a time. The deploy job is outside the group.
    version: 2.1
    
    jobs:
      test:
        docker:
          - image: cimg/base:current
        steps:
          - checkout
          - run: ./scripts/test.sh
      migrate:
        docker:
          - image: cimg/base:current
        steps:
          - checkout
          - run: ./scripts/migrate.sh
      deploy:
        docker:
          - image: cimg/base:current
        steps:
          - checkout
          - run: ./scripts/deploy.sh
    
    workflows:
      main-workflow:
        jobs:
          - test
          - migrate:
              serial-group: shared-db/migrations
              requires:
                - test
          - deploy:
              requires:
                - migrate

Notes

  • The lock is released as soon as the migrate job finishes. The deploy job in this example is not part of the group, so another service can start its migration while this service deploys.

  • If a migration fails, CircleCI still releases the lock and the next migration in the queue starts. The deploy job does not run, because it requires migrate to succeed.

  • A newer pipeline in the same project takes priority over an older one. An older pipeline’s migration job is skipped when a newer pipeline’s job is waiting or running in the group.

Limit a shared test resource to one job at a time

Goal

You may want to give jobs exclusive use of a resource that can serve only one job at a time. Examples include a pool of physical devices or a test environment with limited capacity.

Steps

  • Add the serial-group key to every job that uses the resource, in every project. Name the group after the resource:

    # Jobs from any project that use this serial-group string take turns
    # on the shared device pool.
    version: 2.1
    
    jobs:
      build:
        docker:
          - image: cimg/base:current
        steps:
          - checkout
          - run: ./scripts/build.sh
      device-tests:
        docker:
          - image: cimg/base:current
        steps:
          - checkout
          - run: ./scripts/run-device-tests.sh
    
    workflows:
      main-workflow:
        jobs:
          - build
          - device-tests:
              serial-group: test-resources/device-pool
              requires:
                - build

Notes

  • Use one group for each resource. Jobs that use different resources can run at the same time.

  • If a job holds the lock and is stuck, you can release the group from the web app. See Release a Serial Group.

Run a set of jobs as one unit

Goal

You may want a sequence of jobs, such as a migration followed by a verification step, to finish before any other pipeline starts the same sequence.

Steps

  1. Define the jobs as a job group. Use requires inside the group to set the order of its jobs.

  2. Add the serial-group key where you invoke the group in the workflow:

    # The serial-group key on the group invocation holds the lock until
    # every job in the group has finished.
    version: 2.1
    
    jobs:
      build:
        docker:
          - image: cimg/base:current
        steps:
          - checkout
          - run: ./scripts/build.sh
      migrate:
        docker:
          - image: cimg/base:current
        steps:
          - checkout
          - run: ./scripts/migrate.sh
      verify:
        docker:
          - image: cimg/base:current
        steps:
          - checkout
          - run: ./scripts/verify.sh
      notify:
        docker:
          - image: cimg/base:current
        steps:
          - run: echo "Migration complete"
    
    job-groups:
      migrate-and-verify:
        jobs:
          - migrate
          - verify:
              requires:
                - migrate
    
    workflows:
      main-workflow:
        jobs:
          - build
          - migrate-and-verify:
              serial-group: shared-db/migrations
              requires:
                - build
          - notify:
              requires:
                - migrate-and-verify

Notes

  • The lock is released when all jobs in the group have finished, not after the first job.

  • A job that requires the group, such as notify in this example, runs only if all of the group’s leaf jobs succeeded. See How Job Group Expansion Works.