Changelog

What’s New in Release 2.19.15

Notes

In March 2022 updates to CircleCI server v2.x will stop.

Fixes

• Removed out-dated truststore for root CAs.
• Patched various security vulnerabilities.

Known Issues

• On static (non-AWS) installations with NONE selected under storage driver settings, tests can be split only by file name or file size. If a user attempts to split tests by timing data, static instances will split them by file name instead.
• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds (for further information on SSH reruns see the SSH Rerun Guide):
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if applicable) or private IP of the nomad client

Collapse

What’s New in Release 2.19.14

Notes

In March 2022 updates to CircleCI server v2.x will stop. Support for customers using CircleCI server v2.x will continue.

• There has been a change to how you will checkout using tags in jobs. Branch tags will now be referenced by either CIRCLE_TAG or +refs/heads/tag/path. For example, git checkout --force "$CIRCLE_TAG" or git fetch --force origin '+refs/heads/master:refs/remotes/origin/master'

Fixes

• Security patches for the exim image
• Replaced deprecated GitHub API endpoint for teams

Known Issues

• On static (non-AWS) installations with NONE selected under storage driver settings, tests can be split only by file name or file size. If a user attempts to split tests by timing data, static instances will split them by file name instead.
• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds (for further information on SSH reruns see the SSH Rerun Guide):
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if applicable) or private IP of the nomad client

Collapse

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.3.1

Fixes

• CircleCI runner can now be used when server is installed behind a proxy.
• Fixed a bug that was causing some intermittent broken dependency caches.
• Removed outdated cert trust stores from some images. They were found to be causing access issues to some server installations.
• Fixed a bug that was affecting runner updates.
• SSH reruns now work when server is installed behind a proxy.
• Replaced deprecated GitHub API endoint for teams.

Known Issues

• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised, for example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Collapse

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.3.0

Features

• Nomad Autoscaler can now be used to scale Nomad clients. For more information see the execution environments installation docs.
• Webhooks are now available.
• The Insights dashboard is now available.
• IRSA (AWS) can now be used as an alternative to keys for object storage authentication.
• The email address from which build notifications are sent is now configurable from the KOTS admin console.
• We have replaced Traefik with Kong as our reverse proxy. However, in order to minimize disruption when upgrading, we chose not to rename the service used by Kong. Therefore, you will see a service named circleci-server-traefik, however, this service is actually for Kong.

Fixes

• Upgraded Python to v3 in Vault container.
• Docs improvements on using the shared VPC architecture in GCP.
• The JVM heap size has been updated so that output-processor can use up to 80% of the pod memory limit.

Known Issues

• Updating GitHub Enterprise with a modified Let’s Encrypt certificate is unsupported.
• Let’s Encrypt’s new root certificate isrgrootx1 is not trusted.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised, for example, by using a VPN into your VPC.
• It is currently possible for multiple organizations under the same CircleCI server account to have contexts with identical names. This should be avoided as doing so could lead to errors and unexpected behavior.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.
• Runner cannot be used when server is installed behind a proxy.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Collapse

What’s New in Release 2.19.13

Notes

In March 2022 updates to CircleCI server v2.x will stop. Support for customers using CircleCI server v2.x will continue.

Fixes

• Various security and vulnerability patches.

Known Issues

• On static (non-AWS) installations with NONE selected under storage driver settings, tests can be split only by file name or file size. If a user attempts to split tests by timing data, static instances will split them by file name instead.
• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds (for further information on SSH reruns see the SSH Rerun Guide):
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if applicable) or private IP of the nomad client

Collapse

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.2.2

Notes

• The rerun workflow endpoint now returns workflow ID rather than the message accepted.

Fixes

• TLS is terminated outside of frontend so the SSL server has been completely removed from the frontend container.
• Moved the default certificate logic from KOTS to helm.
• Fixed the build agent image version used in server v3.x. The image used previously was causing problems with runner.

Known Issues

• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised, for example, by using a VPN into your VPC.
• It is currently possible for multiple organizations under the same CircleCI server account to have contexts with identical names. This should be avoided as doing so could lead to errors and unexpected behavior.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.
• Runner cannot be used when server is installed behind a proxy.
• Let’s Encrypt certificate generation does not work. You will need to provide your own certificates or use the default certificates provided.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Collapse

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.2.1

New Features

• Private VMs are now supported for installations on GCP.

Fixes

• mTLS is now disabled by default.
• SSH timeout for VMs has been increased to 10 minutes.
• Private VMs now request private IPs.

Known Issues

• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised, for example, by using a VPN into your VPC.
• It is currently possible for multiple organizations under the same CircleCI server account to have contexts with identical names. This should be avoided as doing so could lead to errors and unexpected behavior.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.
• Runner cannot be used when server is installed behind a proxy.
• Let’s Encrypt certificate generation does not work. You will need to provide your own certificates or use the default certificates provided.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Collapse

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.2.0

New Features

• Customers who require a fully private installation can now access a setting in the KOTS admin console to ensure public IPs are not assigned to VMs. Note that with this non-public IP setting enabled, a work-around will be needed if SSH access to running jobs is required, for example, by using a VPN into your VPC.
• Customers that manage outbound traffic through a proxy can now configure proxy settings through the KOTS admin console. Please see our documentation for specifics on proxy support for server.
• We have expanded the machine execution environment options available to include additional resource classes, sizes, and executors. You now have access to Arm (medium, large), Linux (medium, large, X large, and XX large), and Windows (medium, large, XX large) resource classes.
• The insights API is now available to all server customers. Leverage build and other data to better understand the performance of teams and the health of your build and testing efforts.
• We have revamped the admin UI, and updated our installation instructions, making it easier to set up and manage server.
• You can now supply a custom Linux AMI for VM service.
• SSL termination can now be disabled - If you have put server login behind a firewall, this will enable SSL termination at the firewall.
• You can now control the size of persistent volumes. For larger customers, the initial persistent volume size was too small, by default. You can now set this at install time, providing an easier migration for those customers that require it. For further information see the Internal Database Volume Expansion doc.
• We have added an auto-scaling example to the nomad client terraform.
• You can now choose to serve ‘unsafe’ build artifacts. Previously this option was hidden, meaning potentially unsafe artifacts were rendered as plain text. For more information see the Build Artifacts doc.

Fixes

• The default windows executor was not as documented, we have increased the size to align with documentation and cloud.

Known Issues

• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised, for example, by using a VPN into your VPC.
• It is currently possible for multiple organizations under the same CircleCI server account to have contexts with identical names. This should be avoided as doing so could lead to errors and unexpected behavior.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.
• Runner cannot be used when server is installed behind a proxy.
• Let’s Encrypt certificate generation does not work. You will need to provide your own certificates or use the default certificates provided.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Collapse

Before Upgrading

IMPORTANT: With this release, the frontend-external load balancer has been removed. The traefik load balancer now handles all incoming traffic. When updating from a previous server 3.x version, you will need to update the DNS record that was pointing to the frontend-external load balancer and have it point to the circleci-server-traefik load balancer instead. Remember, you can retrieve the external IP address or DNS name of your traefik load balancer by typing kubectl get svc/circleci-server-traefik in a terminal that has access to the cluster.

For further information see the What’s New doc.

What’s New in Release 3.1.0

New Features

• Telegraf plugins can now be added to server and customized to use third party monitoring solutions, for example, Datadog. For more information, see the Metrics and Monitoring doc.
• The option to use only private load balancers has been introduced for customers who want a fully private installation. For more information see the Load Balancers guide.
• Server 3.x hosts build artifacts, test results, and other state in object storage. We support any S3-compatible storage and Google Cloud Storage. For more information, see the Installation guide for further information.
• Dynamic config via setup workflows is now available on server installations. For more information see our blog post and the Dynamic Configuration docs page.
• Runner is now available on server. For further information, including installation steps, see the Runner docs. Runner allows the use of the macOS executor in server installations and VM service functionality for customers with server installed in a private data centre.
• The frontend load balancer from v3.0 has been removed and replaced with an Ingress resource and the Traefik Ingress controller. This is a breaking change requiring you to reconfigure your DNS. See the What’s New in server docs for further information and guidance.
• The following services can now be externalized. For setup information, see the server v3.x installation guide:
  • Postgres
  • MongoDB
  • Vault
• Backup and restore functionality is now available. For more information see the Backup and Restore guide.
• Prometheus is now deployed by default with server to monitor your cluster health and usage. Prometheus can be managed and configured from the KOTS admin UI. For further information, see the Metrics and Monitoring doc.
• Server now supports the 2XL resource class. The Nomad cluster needs to be made large enough to account for larger resource classes.
• The lifecycle of build artifacts and test results can now be configured from the KOTS admin console under Storage Object Expiry, including the option to disable the expiration and retain artifacts and test results indefinitely.

Fixes

• Resolved a collection of bugs that were causing sensitive information to be leaked into CircleCI support bundles:
  • Instances of faulty and partial redactions of secrets were detected, in part due to 3rd party bugs.
  • PostgresDB leaking sensitive information to STDOUT.
  • Several CircleCI services were logging secrets.
• Tightened network security in the Nomad terraform module.
• Terraform v0.15.0 and up are now supported.
• Updated installation scripts to use functions supported by most recent versions of Terraform.
• Resolved a bug that was leading to machine large builds being run on the wrong machine type. Machine large builds now correctly use 4 vCPUs and 16GB of RAM.
• Resolved a bug that caused contexts-service to fail on expiration of Vault client tokens.
• Resolved a bug that was causing legacy-notifier to report readiness prematurely.
• The JVM heap size parameter has been removed for all services. The heap size is set to be half of the memory limit.
• Changes to networking config and certs are now picked up automatically by Traefik. Previously, a restart would have been required.
• Minimum requirements for CPU and memory have changed. For the new values, see the Installation Prerequisites doc.

Known Issues

• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised, for example, by using a VPN into your VPC.
• It is currently possible for multiple organizations under the same CircleCI server account to have contexts with identical names. This should be avoided as doing so could lead to errors and unexpected behavior.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 3.0 installation, migration, or operations please see our documentation.

Collapse