Start Building for Free
CircleCI.comAcademyBlogCommunitySupport

Container runner reference guide

1 week ago8 min read
Cloud
On This Page

Introduction and motivation

CircleCI’s self-hosted runner has historically executed each job using a one to one mapping between the CI job and a machine environment (virtual or physical) that has the self-hosted runner binary installed on it. By exclusively running jobs in this manner, users sacrifice several benefits of a container-based solution that are afforded on CircleCI’s cloud platform when using the Docker executor:

  • The ability to seamlessly use custom Docker images during job execution.

  • Access to a consistent, clean containerized build environment with every job

Container runner is a type of self-hosted runner a user can install in a Kubernetes (k8s) cluster which enables them to run containerized CI jobs on self-hosted compute, similar to how jobs that use the native Docker executor run on CircleCI’s cloud platform.

Container runner is a complement to the machine runner, not a replacement.

After installation of the container-agent, the container runner will claim your containerized jobs, schedule them within an ephemeral pod, and execute the work within a container-based execution environment.

Container-agent model
Figure 1. Container-agent model - Many container-agent : task-agents

View the Runner FAQ page for a full sample configuration of container runner.

Running your first job with container runner

Follow the instructions outlined on the self-hosted runner installation page to download the container runner and run your first job. You can also use the CircleCI web app to get started with self-hosted runners.

Resource class configuration and custom task pod configuration

Container runner supports claiming and running tasks from multiple resource classes concurrently, as well as customization of the Kubernetes resources created to run tasks for a particular resource class. Configuration is provided by a map object in the Helm chart values.yaml.

Each resource class supports the following parameters:

  • token: The runner resource class token used to claim tasks (required).

  • Custom Kubernetes pod configuration for pods used to run CircleCI jobs.

The pod configuration takes all fields that a normal Kubernetes pod does. If service containers are used in a CircleCI job, the first container spec is used for all containers within the task pod. There is currently no way to provide a different container configuration between service containers and the main task container.

The following fields will be overwritten by container runner to ensure correct task function, and expected CircleCI configuration behavior:

  • spec.containers[0].name

  • spec.containers[0].container.image

  • spec.containers[0].container.args

  • spec.containers[0].container.command

  • spec.containers[0].container.workingDir

  • spec.restartPolicy

  • metadata.name

  • metadata.namespace

Below is a full configuration example, containing two resource classes:

agent:
  resourceClasses:
    circleci-runner/resourceClass:
      token: TOKEN1
      metadata:
        annotations:
          custom.io: my-annotation
      spec:
        containers:
          - resources:
              limits:
                cpu: 500m
            volumeMounts:
              - name: xyz
                mountPath: /path/to/mount
        securityContext:
          runAsNonRoot: true
        imagePullSecrets:
          - name: my_cred
        volumes:
          - name: xyz
            emptyDir: {}

    circleci-runner/resourceClass2:
      token: TOKEN2
      spec:
        imagePullSecrets:
          - name: "other"

Custom token secret

Using the configuration described above provisions a Kubernetes secret containing your resource class tokens. In some circumstances, you may wish to provision your own secret, or you simply might not want to specify the tokens via helm. Instead, you can provision your own Kubernetes secret containing your tokens and specify its name in the agent.customSecret field.

The secret should contain a field for each resource class, using the resource class name as the key and the token as the value. Consider the following resourceClasses configuration:

agent:
  resourceClasses:
    circleci-runner/resourceClass:
      metadata:
        annotations:
          custom.io: <my-annotation>

    circleci-runner/resourceClass2:

The corresponding custom secret would have 2 fields:

circleci-runner.resourceClass: <my-token>
circleci-runner.resourceClass2: <my-token-2>

Due to Kubernetes secret key character constraints, the / separating the namespace and resouce class name is replaced with a . character. Other than this, the name must exactly match the resourceClasses config to match the token with the correct configuration.

Even if there is no further pod configuration, the resource class must be present in resourceClasses as an emtpy map, as shown by circleci-runner/resourceClass2 in the above config example.

Helm Chart Parameters

The following are CircleCI specific settings:

ParameterDescriptionDefault

agent.runnerAPI

Runner API URL

https://runner.circleci.com

agent.name

A (preferably) unique name assigned to this particular container-agent instance. This name will appear in your Runner Inventory page in the CircleCI UI. If left unspecified, the name will default to the name of the deployment.

container-agent (the name of the deployment)

agent.resourceClasses Default must be updated in order to run a job successfully

Resource class task configuration. See the " Resource Class Configuration" section above.

{}

agent.customSecret

A user provided Kubernetes containing resource class tokens. See the " Custom Token Secret" section above.

""

agent.terminationGracePeriodSeconds

Termination grace period during container runner shutdown

18300

agent.maxRunTime

Max task run time. This value should be shorter than the grace period above - See docs for potential values

5h

agent.maxConcurrentTasks

Maximum number of tasks claimed/run concurrently

20

agent.kubeGCEnabled

Option to enabled/disable garbage collection

true

agent.kubeGCThreshold

Length of time pods can run before deleted by GC

5h5m

agent.constraintChecker.enable

Whether to enable the constraint checker

false

agent.constraintChecker.threshold

Number of failed checks before disabling resource class claim

3

agent.constraintChecker.interval

The constraint check interval

15m


The following is for Kubernetes object settings. All settings prefixed with agent below are for the container runner pod itself, not the ephemeral pods where jobs are executed.

ParameterDescriptionDefault

nameOverride

Override the chart name

""

fullnameOverride

Override the full generated name

""

agent.replicaCount

Number of container-agents to deploy. The recommendation is to leave this value at 1

1

agent.image.registry

Agent image registry

""

agent.image.repository

Agent image repository

circleci/container-agent

agent.image.pullPolicy

Agent image pull policy

Always

agent.image.tag

Agent image tag

edge

agent.pullSecrets

Secret objects container private registry credentials for the container runner pod itself, not the ephemeral pods that execute tasks

[]

agent.matchLabels

Match labels used on agent pods

app: container-agent

agent.podAnnotations

Extra annotations added to agent pods

{}

agent.podSecurityContext

Security context policies added to agent pods

{}

agent.containerSecurityContext

Security context policies add to agent containers

{}

agent.resources

Custom resource specifications for container runner pods

{}

agent.nodeSelector

Node selector for agent pods

{}

agent.tolerations

Node tolerations for agent pods

{}

agent.tolerations

Node tolerations for agent pods

[]

agent.affinity

Node affinity for agent pods

{}

agent.autodetectPlatform

Autodetect the OS and CPU architecture of the node that the task pod is running on. If false, the node is assumed to be the same OS and CPU architecture as the container runner pod and cluster-wide permissions are unneeded.

true

serviceAccount.create

Create a custom service account for the agent

true

rbac.create

Create a Role and RoleBinding for the service account

true

logging.image.registry

Image registry for logging containers

""

logging.image.repository

Image repository for logging containers

circleci/logging-collector

logging.image.tag

Image tag for logging containers

edge

logging.serviceAccount.create

Create a custom service account token for logging containers

true

logging.rbac.create

Create a Role and RoleBinding for logging containers

true

Container runner needs the following Kubernetes permissions:

  • Pods, Pods/Exec

    • Get

    • Watch

    • List

    • Create

    • Delete

  • Secrets

    • Get

    • List

    • Create

    • Delete

  • Events

    • Watch

  • Nodes

    • Get

    • List

In addition, Logging containers require the following minimal permissions to get service container logs and stream them to the CircleCI web app:

  • Pods, Pods/Logs

    • Watch

By default a Role, RoleBinding and service account are created and attached to the container runner pod, but if you customize these, the above are the minimum required permissions.

It is assumed that the container runner is running in a Kubernetes namespace without any other workloads. It is possible that the agent or garbage collection (GC) could delete pods in the same namespace.

Garbage collection

Each container runner has a garbage collector which will ensure any pods and secrets with the label app.kubernetes.io/managed-by=circleci-container-agent left dangling in the cluster are removed. By default this will remove all jobs older than five hours and five minutes. This can be shortened or lengthened via the agent.kubeGCThreshold parameter. However, if you do shorten the garbage collection (GC) frequency, also shorten the max task run time via the agent.maxRunTime parameter to be a value smaller than the new GC frequency. Otherwise a running task pod could be removed by the GC.

Container runner will drain and restart cleanly when sent a termination signal. Container runner will not automatically attempt to launch a task that fails to start. This can be done in the CircleCI web app.

If the container runner crashes, there is no expectation that in-process or queued tasks are handled gracefully.

Logging containers

Container runner schedules a logging container if there are service containers in the task pod. This container will get the service container logs and stream them to the CircleCI web app.

Logging containers require a service account token with the minimal privileges to get container logs.

Constraint Validation

Container runner allows you to configure task pods with the full range of Kubernetes settings. This means pods can potentially be configured in a way which cannot be scheduled due to their constraints. To help with this, container runner has a constraint checker which periodically validates each resource class configuration against the current state of the cluster, to ensure pods can be scheduled. This prevents container runner claiming jobs which it cannot schedule which would then fail.

If the constraint checker fails too many checks, it will disable claiming for that resource class until the checks start to pass again.

Currently the following constraints are checked against the cluster state:

As an example of how this works, consider the following resource class configuration:

agent:
  resourceClasses:
    circleci-runner/resourceClass:
      token: TOKEN1
      spec:
        nodeSelector:
          disktype: ssd

    circleci-runner/resourceClass2:
      token: TOKEN2

The first resource class has a node selector to ensure it is scheduled to nodes with an SSD. For some reason during operations the cluster no longer has any nodes with that label. The constraint checker will now fail checks for circleci-runner/resourceClass and will disable claiming jobs until it finds nodes with the correct label again. circleci-runner/resourceClass2 claiming is not affected, the checks for different resource classes are independent of each other.

Cost and availability

Container runner jobs are eligible for Runner Network Egress. This is in line with the existing pricing model for self-hosted runners, and will happen with close adherence to the rest of CircleCI’s network and storage billing roll-out. If there are questions, reach out to your point of contact at CircleCI.

The same plan-based offerings for self-hosted runner concurrency limits apply to the container runner. Final pricing and plan availability will be announced closer to the general availability of the offering.

Building container images

Docker in Docker is not recommended due to the security risk it can pose to your cluster.

To build container images in a container-agent job, a user may use:

  1. A third-party tool like Buildah or Kaniko

  2. Machine runner installed with Docker installed on it

  3. CircleCI-hosted compute

Note: Third-party tools should be used at your own discretion.

While jobs that run with container-agent cannot use CircleCI’s setup_remote_docker feature, it is possible to use a third-party tool to build Docker images in your container-agent job without using the Docker daemon.

You can see an example on our community forum of how some users have successfully used Kaniko to build a container image.

Another option is to use a tool called Buildah. Buildah can be used in your .circleci/config.yml syntax:

docker:
  - image: quay.io/buildah/stable:v1.27.0

Using the Buildah image

Buildah relies on the fuse-overlay program inside of the container, which means that a fuse device plugin must be configured in order to use it. /dev/fuse is required to use fuse-overlayfs inside of the container, as this option tells Buildah on the host to add /dev/fuse to the container for Buildah’s use. Kubernetes has a device plugin system to enable secure sharing of host devices with pods.

To install the configuration dev/fuse, clone this repository to where you are running Helm commands for your container-agent deployment. Then run:

kubectl create -f fuse-device-plugin-k8s-1.16.yml

You can confirm that this has been configured correctly by running kubectl get daemonset -n kube-system and confirming that fuse-device-plugin-daemonset is present and ready.

Once this device has been added, update the container-agent resource class configuration:

resourceClasses:
 <namespace>/<resourceClass>:
  token: <token>
   spec:
    containers:
     - resources:
        limits:
         github.com/fuse: 1

This will now let you run Buildah commands with container agent jobs and build containers:

  docker-image:
    docker:
      - image: quay.io/buildah/stable
    resource_class: <namespace>/<resourceClass>
    steps:
      - checkout
      - run:
          name: sanity-test
          command: |
            buildah version
      - run:
          name: Building-a-container
          command: |
            buildah bud -f ./Dockerfile -t myimage:0.1
            buildah push myimage:tag

Using Buildah with custom images

You can also build your own custom image and include the installation of Buildah in your Dockerfile:

sudo yum install buildah

If you plan to use a CircleCI convenience image, ensure you add the repository for installation to your job’s steps:

sudo apt-get update
sudo apt-get install -y wget ca-certificates gnupg2
VERSION_ID=$(lsb_release -r | cut -f2)
echo "deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/ /" | sudo tee /etc/apt/sources.list.d/devel-kubic-libcontainers-stable.list
curl -Ls https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/xUbuntu_$VERSION_ID/Release.key | sudo apt-key add -
sudo apt-get update
sudo apt install buildah -y

Additionally, set the isolation variable to default to chroot:

# Default to isolate the filesystem with chroot.
ENV BUILDAH_ISOLATION=chroot

You can then follow the same instructions as Using the Buildah image above to add the fuse device plugin to the container-agent deployment and update your .circleci/config.yml file to use your custom images and build container images in those jobs.

Limitations

  • The ability to rerun a job with SSH.

  • Any known limitation for the existing self-hosted runner will continue to be a limitation of container agent.

  • There is no support for container environments other than Kubernetes at this time.

  • Container runner does not yet work on CircleCI’s server offering

  • setup_remote_docker as a command is not supported with container runner. See Building Container Images.

  • There is a known limitation when using an alpine-based image with microk8s where error code 139 is returned. The recommendation at this time is to build your own image using CircleCI’s base image, and to manually install any additional customizations needed that were present in the alpine-based image. See this Discuss post for more information.

FAQs

Please visit the runner FAQ page to see commonly asked questions about container runner.


Help make this document better

This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.

Need support?

Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.

You can also visit our support site to find support articles, community forums, and training resources.