Container runner installation (Kubernetes)

Prerequisites

To install container runners and run jobs, you will need to have root access, and have the following set up and installed:

Kubernetes 1.25+
Helm 3.x
A Kubernetes namespace without any other workloads
The checkout step configures Git to checkout over SSH - ensure that your cluster is configured to allow outbound connections over port 22 if you wish to use it
The CircleCI CLI if you wish to install runners from the command line

Self-hosted runner terms agreement

1. Create namespace and resource class

2. Container runner installation

Add the container runner Helm repository by running the following command:

helm repo add container-agent https://packagecloud.io/circleci/container-agent/helm

Next, run the following:
```
helm repo update
```
Next, run the following command to create the circleci Kubernetes namespace:
```
kubectl create namespace circleci
```

Create a file called values.yaml containing the following, substituting your namespace and resource class and token:

agent:
  resourceClasses:
    <your-namespace>/<your-resource-class-name>:
      token: <your-resource_class_token>

Finally, run the following command:

helm install container-agent container-agent/container-agent -n circleci -f values.yaml

3. Enable rerun job with SSH (optional) - Open preview

The ability to rerun a job with SSH for CircleCI container runner is in open preview.

To enable this optional feature, follow the instructions below after reading through the considerations. Rerunning jobs with SSH allows you to troubleshoot problems through inspecting log files, running processes, and directory paths.

Retry with SSH considerations

Task-agent runs an embedded SSH server and agent on a dedicated port when the “Rerun job with SSH” option is activated. This feature will not affect any other SSH servers or agents on the cluster that container runner is installed on.
The SSH server is configured for public key authentication. Anyone with permission to initiate a job can rerun it with SSH. However, only the user who initiated the rerun will have their SSH public keys added to the server for the duration of the SSH session.
Rerunning a job with SSH will hold the job open for two hours if a connection is made to the SSH server, or 10 minutes if no connection is made, unless canceled. While in this state, the job is counted against an organization’s concurrency limit, and the task-agent will be unavailable to handle other jobs. Therefore, it is recommended to cancel an SSH rerun job explicitly (through the web UI or CLI) when you have finished debugging.
When establishing a SSH connection, you may experience a warning relating to a host change, due to the reuse of a previously used port to connect to a different pod. RSA host fingerprints can be cross-referenced with the output in the console for verification.

The rerun job with SSH feature makes use of the Gateway API to facilitate SSH access from outside your cluster. To enable this feature, Gateway API resources must first be provisioned on your cluster, and the CircleCI container agent configuration must be updated.

Supported Gateway API implementations: CircleCI has tested, and currently supports Envoy Gateway as an implementation for the Gateway API. Other Gateway API implementations that support TCPRoute resources are likely to be compatible, but not all of these implementations have been tested, and therefore compatibility is not guaranteed. Our recommendation is to use Envoy Gateway as your implementation. Envoy Gateway is currently in beta and is under development.

a. Install Envoy Gateway to your cluster

First, install the Gateway API CRDs and Envoy Gateway, as defined in the Envoy Gateway Helm installation document. To do this, replace <version> with the most recent stable release compatible with your cluster, then run the following command:
```
helm install eg oci://docker.io/envoyproxy/gateway-helm --version <version> -n envoy-gateway-system --create-namespace
```

Finally, wait for Envoy Gateway to become available:

kubectl wait --timeout=5m -n envoy-gateway-system deployment/envoy-gateway --for=condition=Available

b. Enable the Rerun job with SSH feature

Once all Gateway API prerequisites are installed and available, add the agent.ssh.enabled = true parameter to the container runner configuration:
```
agent:
  ssh:
    enabled: true
```
For a full list of SSH configuration options, refer to the Helm chart values.

Redeploy the updated manifest:

helm upgrade --wait --timeout=5m <release-name> container-agent/container-agent -n <namespace> -f values.yaml

Wait for the SSH Gateway to be programmed:
```
kubectl wait gateway --timeout=5m --all --for=condition=Programmed -n <namespace>
```
Container runner is now ready for rerunning jobs with SSH.

Container runner configuration example

Once you have installed configuration runner, select Continue in the CircleCI web app and you will be presented with an example configuration snippet showing a job configured to use your new self-hosted runner resource class.

Once you have installed the container runner within your cluster, create and trigger a CircleCI job that uses the Docker executor to validate the installation. The fields you must set for a specific job to run using your container runners are:

image:
resource_class: <namespace>/<resource-class>

Simple example of how you could set up a job (cimg/base:2021.11 is a commonly used CircleCI Docker image):

version: 2.1

jobs:
  build:
    docker:
      - image: cimg/base:2021.11
    resource_class: <namespace>/<resource-class>
    steps:
      - checkout
      - run: echo "Hi I'm on Runners!"

workflows:
  build-workflow:
    jobs:
      - build

Do not use an existing job that uses setup_remote_docker (see Building container images for more information).

Troubleshooting

Refer to the Troubleshoot Container Runner section of the Troubleshoot Self-hosted Runner guide if you encounter issues installing or running container runner.