Changelog

Machine Runner:

• You can now set runner.ssh.advertise_addr_url instead of a fixed runner.ssh.advertise_addr. The runner fetches the SSH rerun address from that URL when a task starts. Use this when something outside the runner assigns the address. The URL must return a host:port.

Container Runner:

• Fixed a leak that occurred when a task was stopped for exceeding its maximum run time. The runner kept watching the task’s pod after that pod had been deleted, leaving a watcher behind for the life of the agent process. An agent that reached the limit repeatedly could accumulate them.

Both Machine and Container Runner:

• Fixed enforcement of the maximum run time (max_run_time, default 5 hours). Two parts of the runner waited on the same internal timeout signal and only one of them could receive it, so the deadline was sometimes missed. A task that overran was then left to run to completion, and its job could still be reported as successful. The same race could also report a task that had genuinely failed as a success. Tasks that reach the limit are now marked failed and stopped as intended.

Machine Runner:

• Machine runner now passes CircleCI’s built-in environment variables through to command_prefix wrapper scripts, so a wrapper can read job details such as the build number and branch.

Container Runner:

• Fixed garbage collection so that when one agent replica dies, the surviving replicas clean up the task pods it left behind. Previously those orphaned pods could linger and keep reserving cluster resources.

Both Machine and Container Runner:

• The Lifecycle step output now includes the runner’s IP address and a clearer runner name, making it easier to identify which runner picked up a job.

Machine Runner:

• Fixed a bug where a runner with a cached task-agent binary could fail every job with an “unsupported platform” error until the process was restarted. The fix ensures newer task-agent versions are still fetched on demand when available.

Container Runner:

• You can now override the container entrypoint per resource class in your runner configuration. See the documentation for details.

Container Runner:

• Fixed an issue where the logging collector exited with a non-zero status on clean shutdown, causing successful jobs to be reported as failed.
• Task pods are now labeled when unsafe retries are enabled.
• Tasks are now only unclaimed after cleanup completes successfully, preventing rescheduling onto an unprepared environment.
• Resource class definitions can now be hot-reloaded from the task pod ConfigMap without requiring a container agent restart.

Machine Runner:

• Fixed an issue where a relative working_directory in config resolved from the task agent’s directory instead of the runner’s, causing steps to run in the wrong location.

Runner Release 3.1.7

Both Machine and Container Runner:

Addressed CVEs in dependencies of the runner agent binary.

Container Runner:

The orchestrator init container’s resource requirements are now configurable via orchestrator.resources in the Helm values (chart version v101.1.6+).

Runner Release 3.1.5

Container Runner:

• Version 3.1.5 introduces Windows support for container runner in open preview. For detailed setup instructions and configuration guidance, visit our Windows container runner documentation.
• Container runner now filters for pods specifically managed by each replica, rather than watching all task pods with the managed-by: circleci-container-agent label. This optimization reduces API load when running multiple container agent instances within the same cluster.
• Added support for running the primary container under a custom user ID (UID). You can now specify a valid UID using the docker.user key in your CircleCI configuration, which will be mapped to the task pod container user.

Runner Release 3.1.0

Container Runner:

• Version 3.1.0 introduces substantial performance and reliability improvements to container runner. These updates are designed to enhance the overall efficiency and stability of operations at scale. For more background and technical details on these changes, you can visit our Runner Init repository and documentation.
• To upgrade, ensure that the Helm chart is at least version v101.1.2. The chart is pinned to the major version tag of the runner, enabling automatic upgrades on redeployment. If you wish to remain on the existing architecture (3.0.x), you can do so by setting the tag to kubernetes-3.0. This version will continue to receive only critical bug fixes and security patches. You can check the configuration details in the Helm chart repository.

NOTE: Release 3.0.23 has been removed from our package repository. It contained bug breaking OIDC token injection for Cloud Runner users. Please use this release 3.0.24 instead.

Container Runner

• Added support for customizable service containers, allowing users more control over a service’s resource usage on a per-image basis. For additional details, refer to the docs.
• Introduced configuration validation to avoid setting the garbage collection threshold below the max task runtime, preventing premature Pod removal and task interruptions.
• Modified the task agent directory to avoid conflicts with existing directories in the PATH variable.

Both Machine and Container Runner:

• Added injection of the CIRCLE_URL variable into task environments. This will resolve to https://runner.circleci.com on cloud, or to the host URL of your server installation on enterprise server.

For customers trying to find cost efficiencies in their runner environment we now support spot instances. The feature can also improve reliability when there are unexpected failures. It comes with some risks and we have documented risks and usage. This release also includes some enhancements and security fixes.

Container Runner:

• Support for unsafe retries, which enable container-agent to automatically rerun tasks that get interrupted during their execution. For example, unsafe retries can be used to run idempotent workloads on spot instances for cost-saving benefits without the worry of Pod preemption causing unexpected job failures. For important information and risk assessment on setting up and using unsafe retries, please read the docs: https://circleci.com/docs/container-runner/#unsafe-retries
• Changed the format of the arbitrary suffix appended to the names of Kubernetes objects that are managed by container-agent. This change has no operational impact on runner.

Both Machine and Container Runner:

• The CircleCI CLI is now prioritized over task agent on the $PATH. That is, invoking the circleci command within a task will point to the CLI binary, if present.
• Patched with Go v1.21.10 for security fixes in Go: https://groups.google.com/g/golang-announce/c/wkkO4P9stm0

Note: If you are running CircleCI Server, you will need version 4.4.3 of server to take advantage of these runner features.

Container Runner

• Reduce long term memory footprint of Container Runner
• Retry tasks where the pod has failed to start due to pod startup errors
• Improve Container Runner monitoring mechanisms for active task pods
• Use the home SSH directory for checkout keys instead of a temporary directory

Machine Runner

• Optionally use the user’s home SSH directory for checkout keys instead of a temporary directory

Critical security fix

• Patched with Go v1.21.5 for critical security fixes in Go. Go release announcement: https://groups.google.com/g/golang-announce/c/iLGK3x6yuNo

Enhancements to error logging and event handling:

• Now logging the outcome of the task Pod’s PreStop hook

• Clearer and actionable error messaging when the task Pod fails to startup or its liveness probe fails

General improvements

• The garbage collector period is now configurable for tuning how quickly failed Pods are removed

• Ability to customize individual values (e.g., the timeoutSeconds) on the task Pod’s primary container’s liveness probe. Any overrides are merged with the defaults, like a kubectl patch.

• See runner configuration documentation for all configuration options