Start Building for Free

CircleCI Server v2.19.x What’s New

4 months ago5 min read
Server v2.x
Server Admin
On This Page

This document provides a summary of features and product notes for the release of CircleCI server v2.19. For a full list of changes, including patch releases, refer to the changelog. For a step by step guide to upgrading your CircleCI server installation to v2.19.x, see our upgrade guide.

Requirements for Upgrading

Notes and Best Practices

  • We now require a minimum 32GB of RAM for the Services Machine.

  • We made some changes to our Redis configuration in v2.18. If you have externalized Redis then you will need to update your configuration. Please contact your Customer Success Manager if you are upgrading from pre v2.18 to v2.19.

  • We have made changes to our Postgres version and require at least postgreSQL v9.5.16. If you have externalized postgreSQL then please update to at least that version in 2.17.x before upgrading to v2.19.

  • There has been a change to how you will checkout using tags in jobs. Branch tags will now be referenced by either CIRCLE_TAG or +refs/heads/tag/path. For example, git checkout --force "$CIRCLE_TAG" or git fetch --force origin '+refs/heads/master:refs/remotes/origin/master'

Known Issues

  • On static (non-AWS) installations with NONE selected under storage driver settings, tests can be split only by file name or file size. If a user attempts to split tests by timing data, static instances will split them by file name instead.

  • If you are using an HTTP proxy for your installation, upgrade to v2.19.02 or above. We had a known issue around job step logging and docker commands customization that made pre-2.19.2 releases incompatible with an HTTP proxy setup of CircleCI Server.

  • The first user to access a CircleCI Server installation should be granted admin access. Currently, there is an issue preventing this from happening when LDAP authentication is in use. Follow this guide for a workaround.

  • If any changes have been made to your networking configuration from the default, you may need to run the following steps to ensure you can use SSH to inspect your builds:

    • On all Nomad Client machine, create /etc/circleci/public-ipv4

    • This file should contain the public (if applicable) or private IP of the nomad client

  • Classic Load Balancer is no longer available from v2.19.02 due to the ciphersuite changes listed in the notes below. CircleCI no longer accepts requests from Classic Load Balancer, so you should move to Network Load Balancer (NLB) or Application Load Balancer (ALB).

What’s New in v2.19.15

  • Removed out-dated truststore for root CAs.

  • Patched various security vulnerabilities.

What’s New in v2.19.14

  • Security patches for the exim image.

  • Replaced deprecated GitHub API endpoint for teams.

What’s New in v2.19.13

  • Various security and vulnerability patches.

What’s New in v2.19.12

  • Removed support for 1.0 builder graceful shutdown on auto-scaling events.

    • This alleviates Python 2 security vulnerabilities.

What’s New in v2.19.11

  • Fixed a bug that didn’t allow more than 50 aliases in a configuration.

What’s New in v2.19.10

  • Fixed a bug that caused 'git checkout' to fail in go-git.

  • Fixed a bug in the YAML parser in picard dispatcher to support config-str to have more than 50 YAML aliases.

  • Fixed a bug where s3 retention policy deleted test results from an older job.

  • Reduced the likelihood of 'out-of-sequence' commit statuses.

  • General security updates.

What’s New in v2.19.09

  • Better handling of non-alphanumeric characters in authentication passwords for Docker executors. Fixed a bug that was preventing the use of ~ and ? in the Docker ID password field under the auth key.

  • General security updates.

What’s New in v2.19.08

  • Fixed a bug that was causing parallelism to fail for static installations with NONE selected under storage driver settings.

What’s New in v2.19.07

  • Fixed a bug that was preventing authentication to China region ECR images in the docker executor.

  • Removed the recursive chown from startup of the fileserverd service. In instances of heavy usage this was causing the startup process to take a long time, or in some cases, startup was blocked.

What’s New in v2.19.06

  • Fixed a bug that was causing workflow statuses to be displayed on GitHub in the wrong order.

  • Introduced performance improvements to workflows-conductor that dramatically reduce CPU usage and latency.

  • Fixed a bug that was preventing use of an S3 storage region other than us-east-1 via an IAM user.

  • Fixed formatting type of SMTP passwords to ensure they are masked during setup.

What’s New in v2.19.05

  • Fixed a bug that could lead to the VM database ending up in an incorrect state in the event of a Services Machine crash or manual termination of VM Service instances.

What’s New in v2.19.04

  • Fixed a bug that was leading to support bundle creation timing out before the Services machine logs had been created, leaving only Replicated logs.

  • S3 connection pool metrics under circle.s3.connection_pool.* have been added to the test results service, making it easier to debug issues with this service.

  • Added in missing environment variables for the workflows service. The absence of these environment variables was causing excessive stack tracing whenever workflows were run. This in turn lead to excessive log rotation.

  • Fixed a bug that was causing failure to update GitHub statuses. Some customers experienced this bug when a project had a follower with a broken auth token.

What’s New in v2.19.03

  • Removed the use of the depecated API endpoint GET applications/%s/tokens/%s.

  • Distributed tracing is now enabled by default for Server installations. Traces are used in CircleCI support bundles to improve our ability to troubleshoot Server issues. Options for the tracing sampling rate are displayed in the Replicated Management Console, but should only be changed from the default if requested by CircleCI Support.

  • Fixed an issue that was preventing restore_cache from working with the storage driver set to "none" - i.e not S3.

  • Fixed an issue that was preventing the output_processor service from using AWS AssumeRole when the role was located in a subfolder. This issue affected customers with security policies forcing the use of a subfolder in this case, and the symptoms included the inability to store artifacts or use timings-based test splitting.

  • JVM heap size can now be changed using the JVM_HEAP_SIZE environment variable for the following services: vm-service, domain-service, permissions-service and federations-service.

What’s New in v2.19.02

  • In the LDAP login flow we now use an anonymous form to POST LDAP auth state, rather than sending it as a GET parameter. Previously, when a user authenticated using LDAP, their username and password were sent in plaintext as part of a query parameter in a GET request. As requests are over HTTPS, this left usernames and passwords in request logs, etc. This issue is now fixed.

  • Optimizely and Zendesk are now removed from Server release images.

  • Fixed an issue in which setting CIRCLE_ADMIN_SERVER_HTTP_THREADS or CIRCLE_PUBLIC_FACING_SERVER_HTTP_THREADS too high would prevent the frontend container from starting.

  • Due to changes in the GitHub API we have removed the use of ?client_id=x&client_secret=y for GitHub, and GHE versions 2.17 and later.

  • Fixed an issue that was causing intermittent failures to spin up VMs with DLC in use.

  • Fixed an issue that was preventing the customization of proxy settings for Docker containers. See the Nomad Client Proxy and Service Configuration Overrides guides for more infomation.

  • Fixed a bug that was preventing job steps for non-failing builds being logged when proxy settings were used for the job container.

  • Removed legacy TLS versions 1.0 and 1.1, in addition, enabled 1.2 and 1.3 TLS, and specified the following ciphersuites


  • Fixed a statsd configuration issue that meant some services were not emitting Telegraf metrics.

What’s New in v2.19.01

  • Fixed a bug that was preventing some customers from upgrading due to a schema change in one of our library dependencies.

  • Fixed a bug that was preventing some customers from inspecting builds via SSH due to a logic change in our build agent.

What’s New in v2.19

Help make this document better

This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.

Need support?

Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.

You can also visit our support site to find support articles, community forums, and training resources.