서버 변경 로그

Before Upgrading

See the CircleCI server 4.1 release notes for upgrade notes for this release.

When upgrading from server v3.x you will need to install v4.0 before installing this version.

What’s New in Release 4.1.3

Fixes

• Query parameters are now supported on the task-agent download endpoint.

Known Issues

• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.1 installation, migration, or operations please see our documentation.

Before Upgrading

See the CircleCI server 4.1 release notes for upgrade notes for this release.

When upgrading from server v3.x you will need to install v4.0 before installing this version.

What’s New in Release 4.1.2

Changes

• We have updated our PostgreSQL password requirements to allow special characters (except single or double quotes) in addition to alphanumeric characters.

Fixes

• For runner installations on server, you can now directly clone a repository with just the checkout step in .circleci/config.yml.

Known Issues

• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.1 installation, migration, or operations please see our documentation.

Before Upgrading

See the CircleCI server 4.1 release notes for upgrade notes for this release.

When upgrading from server v3.x you will need to install v4.0 before installing this version.

What’s New in Release 4.1.1

Changes

• In the checkout step in a CircleCI configuration, GitHub host keys are appended to ~/.ssh/known_hosts. Due to a change to GitHub’s RSA SSH key, CircleCI have updated the appended key to match. If you are using GitHub.com, and your installation of server has the old host key hardcoded into any projects, you will also need to update these to match.

Known Issues

• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.1 installation, migration, or operations please see our documentation.

Before Upgrading

See the CircleCI server 4.1 release notes for upgrade notes for this release.

When upgrading from server v3.x you will need to install v4.0 before installing this version.

What’s New in Release 4.1.0

New features

• Air-gapped installation is now supported on server v4.1.
• Self-signed certificates or private certificate authorities are now supported when using GitHub Enterprise.
• For new installations, Tink can be used as the encryption backend instead of Vault.

Changes

• The minimum Telegraf version is now set to 1.8.14.
• Nomad and Vault images are now stored in templates, rather than hardcoded in charts.
• The Nomad Terraform configurations now have the Nomad port hardcoded. Previously this needed to be entered manually.

Fixes

• The Trigger Pipeline button in the web app now works for forked PRs, and an error message will be visible if the trigger fails.
• Fixed a bug that was preventing forked PRs from building if both the fork and original were building on CircleCI.
• Fixed a bug that was causing intermittent status updates for jobs in GitHub.

Known Issues

• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.x installation, migration, or operations please see our documentation.

Before upgrading

See the CircleCI server 4.0 release notes for upgrade notes for this release.

What’s new in release 4.0.4

Changes

• We have updated our PostgreSQL password requirements to allow special characters (except single or double quotes) in addition to alphanumeric characters.

Fixes

• For runner installations on server, you can now directly clone a repository with just the checkout step in .circleci/config.yml.

Known issues

• When migrating from 3.x to 4.x, workflow configuration migrations will run as an init job and the workflows-conductor-event-consumer and the workflows-conductor-grpc-handler pods will not start until the migration completes. Customers with large PostgreSQL databases should plan accordingly. Migration time can be reduced by scaling the workflow-conductor pods.
• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.0 installation, migration, or operations please see our documentation.

Before upgrading

See the CircleCI server 4.0 release notes for upgrade notes for this release.

What’s new in release 4.0.3

Changes

• In the checkout step in a CircleCI configuration, GitHub host keys are appended to ~/.ssh/known_hosts. Due to a change to GitHub’s RSA SSH key, CircleCI have updated the appended key to match. If you are using GitHub.com, and your installation of server has the old host key hardcoded into any projects, you will also need to update these to match.

Known issues

• When migrating from 3.x to 4.x, workflow configuration migrations will run as an init job and the workflows-conductor-event-consumer and the workflows-conductor-grpc-handler pods will not start until the migration completes. Customers with large PostgreSQL databases should plan accordingly. Migration time can be reduced by scaling the workflow-conductor pods.
• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.0 installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.4.7

Fixes

• When using CircleCI runner, a repository can now be directly cloned with the checkout step in .circleci/config.yml.

Known Issues

• webhook-service is not operational when Vault is externalized.
• Vault may not refresh its client token after a month of uptime.
• Before upgrading to v3.4.x, delete the deployment named circleci-server-kube-state-metrics.
• Prometheus is broken in KOTS versions 1.65.0 - 1.67.0. If you rely on this feature, do not upgrade your KOTS version until this issue has been resolved.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installations you would need to ensure that you can access the private IP advertised (for example, by using a VPN into your VPC).
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.4.6

Changes

• In the checkout step in a CircleCI configuration, GitHub host keys are appended to ~/.ssh/known_hosts. Due to a change to GitHub’s RSA SSH key, CircleCI have updated the appended key to match. If you are using GitHub.com, and your installation of server has the old host key hardcoded into any projects, you will also need to update these to match.

Known Issues

• webhook-service is not operational when Vault is externalized.
• Vault may not refresh its client token after a month of uptime.
• Before upgrading to v3.4.x, delete the deployment named circleci-server-kube-state-metrics.
• Prometheus is broken in KOTS versions 1.65.0 - 1.67.0. If you rely on this feature, do not upgrade your KOTS version until this issue has been resolved.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installations you would need to ensure that you can access the private IP advertised (for example, by using a VPN into your VPC).
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.4.5

Fixes

• Patched some CVEs.
• Updated runner’s launch-agent to use a query string instead of a GET body.

Known Issues

• webhook-service is not operational when Vault is externalized.
• Vault may not refresh its client token after a month of uptime.
• Before upgrading to v3.4.x, delete the deployment named circleci-server-kube-state-metrics.
• Prometheus is broken in KOTS versions 1.65.0 - 1.67.0. If you rely on this feature, do not upgrade your KOTS version until this issue has been resolved.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installations you would need to ensure that you can access the private IP advertised (for example, by using a VPN into your VPC).
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

End of service notice

CircleCI server v2 will cease to function at the end of March 2023. Server v2 reached EOL April 1, 2022, and has not been supported since that time. We encourage customers still migrating off v2 to reach out to your Customer Service teams.

Before upgrading

See the CircleCI server 4.x release notes for upgrade notes for this release.

What’s new in release 4.0.2

Changes

• Pin Bitnami Postgres Helm chart url

Known issues

• When migrating from 3.x to 4.x, workflow configuration migrations will run as an init job and the workflows-conductor-event-consumer and the workflows-conductor-grpc-handler pods will not start until the migration completes. Customers with large PostgreSQL databases should plan accordingly. Migration time can be reduced by scaling the workflow-conductor pods.
• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.x installation, migration, or operations please see our documentation.

Before upgrading

See the CircleCI server 4.x release notes for upgrade notes for this release.

What’s new in release 4.0.1

Changes

• When migrating to server v4.x, Telegraf configuration data is now decoded and available in your values.yaml file.
• Anonymous statistics reporting for Kong is now set to disabled by default.
• TLS settings are now available for Postgres.

Fixes

• Fixed a bug that was preventing email notifications from being sent with the following setting enabled: Notifications > Email Notifications> Build Notifications > My Branches.
• Fixed a bug that prevented webhook-service from working when Vault is externalized.

Known issues

• When migrating from 3.x to 4.x, workflow configuration migrations will run as an init job and the workflows-conductor-event-consumer and the workflows-conductor-grpc-handler pods will not start until the migration completes. Customers with large PostgreSQL databases should plan accordingly. Migration time can be reduced by scaling the workflow-conductor pods.
• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.x installation, migration, or operations please see our documentation.

CircleCI server v4.0.0 is now generally available. The feature set for this release are equal to server v3.x. The installation and upgrade processes for v4.x have changed significantly.

Server v4.x offers greatly improved security handling, installation, and update processes. Server v4.x is installed using helm charts and images that can be pulled ahead of time to comply with your security processes. Installation processes can also use artifacts pulled from customer-managed registries. Server v4.x makes improved use of Kubernetes secrets and removes the requirement to grant permissions and network access to third-party tools.

Before upgrading

See the CircleCI server 4.x release notes for upgrade notes for this release.

What’s new in release 4.0.0

New features

• Two new Arm resource classes are now available xlarge and 2xlarge. For more information on using Arm, see the Arm docs.
• CircleCI server v4.x images and helm charts are available for licenced users.

Fixes

• Fixed a bug that was causing a huge quantity of error logs in the distributer-internal pod.
• Prometheus is no longer included by default in CircleCI server v4.x.
• The default concurrency limit has been increased to avoid it being reached.
• Docker layer caching volumes are now deleted after 3 days if unused. Previously they were deleted after 7 days.
• kong is now placed behind NGINX to simplify loadbalancer setup.
• The loadbalancer has been renamed from circleci-traefik to circleci-proxy.
• Workflow configurations that are stored in PostgreSQL will migrated to the Object Storage.

Known issues

• Telegraf needs to have its configuration overwritten and base64 encoded.
• When migrating from 3.x to 4.x, workflow configuration migrations will run as an init job and the workflows-conductor-event-consumer and the workflows-conductor-grpc-handler pods will not start until the migration completes. Customers with large PostgreSQL databases should plan accordingly. Migration time can be reduced by scaling the workflow-conductor pods.
• Webhook-service may not work with an externalized Vault.
• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.x installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.4.4

Fixes

• Fixed a bug that was causing delays and failures to update git commit statuses.

Known Issues

• webhook-service is not operational when Vault is externalized.
• Vault may not refresh its client token after a month of uptime.
• Before upgrading to v3.4.x delete the deployment named circleci-server-kube-state-metrics.
• Prometheus is broken in KOTS versions 1.65.0 - 1.67.0. If you rely on this feature, do not upgrade your KOTS version until this issue has been resolved.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installations you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.4.3

Changes

• Anonymous statistics reporting for Kong is now set to disabled by default.

Fixes

• Fixed a bug that was preventing email notifications from being sent with the following setting enabled: Notifications > Email Notifications> Build Notifications > My Branches.
• Fixed a bug that was causing a “Failed to load webhooks” error on project Webhooks pages in the CircleCI web app.

Known Issues

• webhook-service is not operational when Vault is externalized.
• Vault may not refresh its client token after a month of uptime.
• Before upgrading to v3.4.x delete the deployment named circleci-server-kube-state-metrics.
• Prometheus is broken in KOTS versions 1.65.0 - 1.67.0. If you rely on this feature, do not upgrade your KOTS version until this issue has been resolved.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.4.2

Fixes

• Links and redirects within the CircleCI server web app now correctly send users to their server login page. Previously, these links were directing people to the CircleCI cloud login page.
• Improvements to the user experience when upgrading and migrating server versions.
• Root volumes created by VM service are now tagged with the custom tag set for instances in the KOTS admin console. This covers machine executor and remote Docker VMs.
• A bug was fixed that was presenting users who need to re-autorize with a 404.
• CircleCI cloud marketing items have been removed.

Known Issues

• Vault may not refresh its client token after a month of uptime.
• Before upgrading to v3.4.x delete the deployment named circleci-server-kube-state-metrics.
• Prometheus is broken in KOTS versions 1.65.0 - 1.67.0. If you rely on this feature, do not upgrade your KOTS version until this issue has been resolved.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.4.1

Fixes

• Kong resource limits have been increased. The previous limits were affecting the stability of some server installations.
• The KOTS config has been updated to allow private loadbalancers when ACM is in use.
• Fixed a bug that was preventing distributor-external from running in certain environments.

Known Issues

• Before upgrading to v3.4.x delete the deployment named circleci-server-kube-state-metrics.
• Prometheus is broken in KOTS versions 1.65.0 - 1.67.0. If you rely on this feature, do not upgrade your KOTS version until this issue has been resolved.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.4.0

New Features

• Android images for the machine executor are now available. For more information, see the Using Android Images with the Machine Executor guide.
• The CircleCI Build Agent can now be pulled from a custom Docker registry if required. For more information contact customer support.
• You can now use Amazon Certificate Manager (ACM) to manage your TLS certificates. For more information see the installation docs.
• Kubernetes v1.16 - v1.23 are now supported.
• Workload Identities for installations on GCP are now supported. Workload Identities can be used as an alternative to static credentials. For more informations see the installation docs

Fixes

• Fixed a bug that was causing the jobs page and step output access to run very slow in the CircleCI web app.
• Fixed a bug that was causing the workflows conductor pod to crash and issues with StatsD metrics.

Known Issues

• Before installing v3.4.0 delete the deployment named circleci-server-kube-state-metrics.
• Prometheus is broken in KOTS versions 1.65.0 - 1.67.0. If you rely on this feature, do not upgrade your KOTS version until this issue has been resolved.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.
• Config files will now be stored in the chosen object store (e.g., AWS S3, Google Cloud Storage or Minio). Users can run into errors accessing certain pipelines, if these config files are deleted due to retention policies (e.g., AWS S3 Lifecycle).

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.3.1

Fixes

• CircleCI runner can now be used when server is installed behind a proxy.
• Fixed a bug that was causing some intermittent broken dependency caches.
• Removed outdated cert trust stores from some images. They were found to be causing access issues to some server installations.
• Fixed a bug that was affecting runner updates.
• SSH reruns now work when server is installed behind a proxy.
• Replaced deprecated GitHub API endoint for teams.

Known Issues

• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised, for example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.3.0

Features

• Nomad Autoscaler can now be used to scale Nomad clients. For more information see the execution environments installation docs.
• Webhooks are now available.
• The Insights dashboard is now available.
• IRSA (AWS) can now be used as an alternative to keys for object storage authentication.
• The email address from which build notifications are sent is now configurable from the KOTS admin console.
• We have replaced Traefik with Kong as our reverse proxy. However, in order to minimize disruption when upgrading, we chose not to rename the service used by Kong. Therefore, you will see a service named circleci-server-traefik, however, this service is actually for Kong.

Fixes

• Upgraded Python to v3 in Vault container.
• Docs improvements on using the shared VPC architecture in GCP.
• The JVM heap size has been updated so that output-processor can use up to 80% of the pod memory limit.

Known Issues

• Updating GitHub Enterprise with a modified Let’s Encrypt certificate is unsupported.
• Let’s Encrypt’s new root certificate isrgrootx1 is not trusted.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised, for example, by using a VPN into your VPC.
• It is currently possible for multiple organizations under the same CircleCI server account to have contexts with identical names. This should be avoided as doing so could lead to errors and unexpected behavior.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.
• Runner cannot be used when server is installed behind a proxy.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.2.2

Notes

• The rerun workflow endpoint now returns workflow ID rather than the message accepted.

Fixes

• TLS is terminated outside of frontend so the SSL server has been completely removed from the frontend container.
• Moved the default certificate logic from KOTS to helm.
• Fixed the build agent image version used in server v3.x. The image used previously was causing problems with runner.

Known Issues

• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised, for example, by using a VPN into your VPC.
• It is currently possible for multiple organizations under the same CircleCI server account to have contexts with identical names. This should be avoided as doing so could lead to errors and unexpected behavior.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.
• Runner cannot be used when server is installed behind a proxy.
• Let’s Encrypt certificate generation does not work. You will need to provide your own certificates or use the default certificates provided.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.2.1

New Features

• Private VMs are now supported for installations on GCP.

Fixes

• mTLS is now disabled by default.
• SSH timeout for VMs has been increased to 10 minutes.
• Private VMs now request private IPs.

Known Issues

• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised, for example, by using a VPN into your VPC.
• It is currently possible for multiple organizations under the same CircleCI server account to have contexts with identical names. This should be avoided as doing so could lead to errors and unexpected behavior.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.
• Runner cannot be used when server is installed behind a proxy.
• Let’s Encrypt certificate generation does not work. You will need to provide your own certificates or use the default certificates provided.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Before Upgrading

See the What’s new in server 3.x doc for upgrade notes for this release.

What’s New in Release 3.2.0

New Features

• Customers who require a fully private installation can now access a setting in the KOTS admin console to ensure public IPs are not assigned to VMs. Note that with this non-public IP setting enabled, a work-around will be needed if SSH access to running jobs is required, for example, by using a VPN into your VPC.
• Customers that manage outbound traffic through a proxy can now configure proxy settings through the KOTS admin console. Please see our documentation for specifics on proxy support for server.
• We have expanded the machine execution environment options available to include additional resource classes, sizes, and executors. You now have access to Arm (medium, large), Linux (medium, large, X large, and XX large), and Windows (medium, large, XX large) resource classes.
• The insights API is now available to all server customers. Leverage build and other data to better understand the performance of teams and the health of your build and testing efforts.
• We have revamped the admin UI, and updated our installation instructions, making it easier to set up and manage server.
• You can now supply a custom Linux AMI for VM service.
• SSL termination can now be disabled - If you have put server login behind a firewall, this will enable SSL termination at the firewall.
• You can now control the size of persistent volumes. For larger customers, the initial persistent volume size was too small, by default. You can now set this at install time, providing an easier migration for those customers that require it. For further information see the Internal Database Volume Expansion doc.
• We have added an auto-scaling example to the nomad client terraform.
• You can now choose to serve ‘unsafe’ build artifacts. Previously this option was hidden, meaning potentially unsafe artifacts were rendered as plain text. For more information see the Build Artifacts doc.

Fixes

• The default windows executor was not as documented, we have increased the size to align with documentation and cloud.

Known Issues

• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised, for example, by using a VPN into your VPC.
• It is currently possible for multiple organizations under the same CircleCI server account to have contexts with identical names. This should be avoided as doing so could lead to errors and unexpected behavior.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.
• Runner cannot be used when server is installed behind a proxy.
• Let’s Encrypt certificate generation does not work. You will need to provide your own certificates or use the default certificates provided.

To learn more about Server 3.x installation, migration, or operations please see our documentation.

Before Upgrading

IMPORTANT: With this release, the frontend-external load balancer has been removed. The traefik load balancer now handles all incoming traffic. When updating from a previous server 3.x version, you will need to update the DNS record that was pointing to the frontend-external load balancer and have it point to the circleci-server-traefik load balancer instead. Remember, you can retrieve the external IP address or DNS name of your traefik load balancer by typing kubectl get svc/circleci-server-traefik in a terminal that has access to the cluster.

For further information see the What’s New doc.

What’s New in Release 3.1.0

New Features

• Telegraf plugins can now be added to server and customized to use third party monitoring solutions, for example, Datadog. For more information, see the Metrics and Monitoring doc.
• The option to use only private load balancers has been introduced for customers who want a fully private installation. For more information see the Load Balancers guide.
• Server 3.x hosts build artifacts, test results, and other state in object storage. We support any S3-compatible storage and Google Cloud Storage. For more information, see the Installation guide for further information.
• Dynamic config via setup workflows is now available on server installations. For more information see our blog post and the Dynamic Configuration docs page.
• Runner is now available on server. For further information, including installation steps, see the Runner docs. Runner allows the use of the macOS executor in server installations and VM service functionality for customers with server installed in a private data centre.
• The frontend load balancer from v3.0 has been removed and replaced with an Ingress resource and the Traefik Ingress controller. This is a breaking change requiring you to reconfigure your DNS. See the What’s New in server docs for further information and guidance.
• The following services can now be externalized. For setup information, see the server v3.x installation guide:
  • Postgres
  • MongoDB
  • Vault
• Backup and restore functionality is now available. For more information see the Backup and Restore guide.
• Prometheus is now deployed by default with server to monitor your cluster health and usage. Prometheus can be managed and configured from the KOTS admin UI. For further information, see the Metrics and Monitoring doc.
• Server now supports the 2XL resource class. The Nomad cluster needs to be made large enough to account for larger resource classes.
• The lifecycle of build artifacts and test results can now be configured from the KOTS admin console under Storage Object Expiry, including the option to disable the expiration and retain artifacts and test results indefinitely.

Fixes

• Resolved a collection of bugs that were causing sensitive information to be leaked into CircleCI support bundles:
  • Instances of faulty and partial redactions of secrets were detected, in part due to 3rd party bugs.
  • PostgresDB leaking sensitive information to STDOUT.
  • Several CircleCI services were logging secrets.
• Tightened network security in the Nomad terraform module.
• Terraform v0.15.0 and up are now supported.
• Updated installation scripts to use functions supported by most recent versions of Terraform.
• Resolved a bug that was leading to machine large builds being run on the wrong machine type. Machine large builds now correctly use 4 vCPUs and 16GB of RAM.
• Resolved a bug that caused contexts-service to fail on expiration of Vault client tokens.
• Resolved a bug that was causing legacy-notifier to report readiness prematurely.
• The JVM heap size parameter has been removed for all services. The heap size is set to be half of the memory limit.
• Changes to networking config and certs are now picked up automatically by Traefik. Previously, a restart would have been required.
• Minimum requirements for CPU and memory have changed. For the new values, see the Installation Prerequisites doc.

Known Issues

• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised, for example, by using a VPN into your VPC.
• It is currently possible for multiple organizations under the same CircleCI server account to have contexts with identical names. This should be avoided as doing so could lead to errors and unexpected behavior.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 3.0 installation, migration, or operations please see our documentation.

Release 3.0.2

• Resolved a bug relating to artifacts disappearing after 30 days. The default settings for the artifact retention period have been updated to unlimited, and can be adjusted from the KOTS Admin Console.
• Resolved a bug that made Traefik “unaware” of TLS certificate updates without requiring a manual restart of the Traefik pod. The Traefik pod will now restart automatically after any TLS certificate updates go into effect after the initial post KOTS deployment.
• Resolved a bug in builds-service that was causing pods to crash as a result of running out of memory.

Known Issues

• No support for external data stores (Postgres, Mongo, Vault). This feature will be implemented in a future release.
• It is currently possible for multiple organizations under the same CircleCI server account to have contexts with identical names. This should be avoided as doing so could lead to errors and unexpected behavior. This will be fixed in a future patch release.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• CircleCI currently assigns a public load balancer for the frontend services. For some customers, their infrastructure or security groups won’t allow this. We will provide an optional internal local balancer for the frontend services in a future release.
• Telegraf metrics collection customization is not yet available.

To learn more about Server 3.0 installation, migration, or operations please see our documentation.

Release 3.0.1

• build_agent version value updated, as the previous version was relying on a vulnerable version of PsExec.
• Due to an issue that was causing duplicated checks in GitHub, environment variables for output-processor were reconfigured.
• Adjusted deployment configuration for vm-service to handle out-of-order database migrations managed by Flyway.

Known Issues

• No support for external data stores (Postgres, Mongo, Vault). This feature will be implemented in a future release.
• It is currently possible for multiple organizations under the same CircleCI server account to have contexts with identical names. This should be avoided as doing so could lead to errors and unexpected behavior. This will be fixed in a future patch release.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• CircleCI currently assigns a public load balancer for the frontend services. For some customers, their infrastructure or security groups won’t allow this. We will provide an optional internal local balancer for the frontend services in a future release.
• Telegraf metrics collection customization is not yet available.

To learn more about Server 3.0 installation, migration, or operations please see our documentation.

What’s New in Release 3.0.0

Server 3.0 is now generally available. The newest version of server offers the ability to scale under heavy workloads, all within your own Kubernetes cluster and private network, while still enjoying the full CircleCI cloud experience. Server 3.0 includes the latest CircleCI features, such as orbs, scheduled workflows, matrix jobs, and more. For existing customers interested in migrating from 2.19 to 3.x, contact your customer success manager. Server 3.0 will receive monthly patch releases and quarterly feature releases.

New Features

• New UI
• Orbs for Server
• Pipelines
• Config 2.1
• API 2.0
• Scheduled Workflows
• Matrix Jobs
• And much more…

Known Issues

• No support for external data stores (Postgres, Mongo, Vault). This feature will be implemented in a future release.
• It is currently possible for multiple organizations under the same CircleCI server account to have contexts with identical names. This should be avoided as doing so could lead to errors and unexpected behavior. This will be fixed in a future patch release.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• CircleCI currently assigns a public load balancer for the frontend services. For some customers, their infrastructure or security groups won’t allow this. We will provide an optional internal local balancer for the frontend services in a future release.
• Telegraf metrics collection customization is not yet available.
• Support for GPU builders is not yet available.

To learn more about Server 3.0 installation, migration, or operations please see our documentation.

What’s New in Release 2.19.15

Notes

In March 2022 updates to CircleCI server v2.x will stop.

Fixes

• Removed out-dated truststore for root CAs.
• Patched various security vulnerabilities.

Known Issues

• On static (non-AWS) installations with NONE selected under storage driver settings, tests can be split only by file name or file size. If a user attempts to split tests by timing data, static instances will split them by file name instead.
• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds (for further information on SSH reruns see the SSH Rerun Guide):
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if applicable) or private IP of the nomad client

What’s New in Release 2.19.14

Notes

In March 2022 updates to CircleCI server v2.x will stop. Support for customers using CircleCI server v2.x will continue.

• There has been a change to how you will checkout using tags in jobs. Branch tags will now be referenced by either CIRCLE_TAG or +refs/heads/tag/path. For example, git checkout --force "$CIRCLE_TAG" or git fetch --force origin '+refs/heads/master:refs/remotes/origin/master'

Fixes

• Security patches for the exim image
• Replaced deprecated GitHub API endpoint for teams

Known Issues

• On static (non-AWS) installations with NONE selected under storage driver settings, tests can be split only by file name or file size. If a user attempts to split tests by timing data, static instances will split them by file name instead.
• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds (for further information on SSH reruns see the SSH Rerun Guide):
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if applicable) or private IP of the nomad client

What’s New in Release 2.19.13

Notes

In March 2022 updates to CircleCI server v2.x will stop. Support for customers using CircleCI server v2.x will continue.

Fixes

• Various security and vulnerability patches.

Known Issues

• On static (non-AWS) installations with NONE selected under storage driver settings, tests can be split only by file name or file size. If a user attempts to split tests by timing data, static instances will split them by file name instead.
• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds (for further information on SSH reruns see the SSH Rerun Guide):
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if applicable) or private IP of the nomad client

What’s New in Release 2.19.12

Fixes

• Removed support for 1.0 builder graceful shutdown on auto-scaling events
  • This alleviates Python 2 security vulnerabilities

Known Issues

• On static (non-AWS) installations with NONE selected under storage driver settings, tests can be split only by file name or file size. If a user attempts to split tests by timing data, static instances will split them by file name instead.
• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds (for further information on SSH reruns see the SSH Rerun Guide):
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if applicable) or private IP of the nomad client

What’s New in Release 2.19.11

Fixes

• Fixed a bug that didn’t allow more than 50 aliases in a configuration

Known Issues

• On static (non-AWS) installations with NONE selected under storage driver settings, tests can be split only by file name or file size. If a user attempts to split tests by timing data, static instances will split them by file name instead.
• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds (for further information on SSH reruns see the SSH Rerun Guide):
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if applicable) or private IP of the nomad client

What’s New in Release 2.19.10

Fixes

• Fixed a bug that caused ‘git checkout’ to fail in go-git.
• Fixed a bug in the YAML parser in picard dispatcher to support config-str to have more than 50 YAML aliases.
• Fixed a bug where s3 retention policy deleted test results from an older job.
• Reduced the likelihood of ‘out-of-sequence’ commit statuses.
• General security updates.

Known Issues

• On static (non-AWS) installations with NONE selected under storage driver settings, tests can be split only by file name or file size. If a user attempts to split tests by timing data, static instances will split them by file name instead.
• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds (for further information on SSH reruns see the SSH Rerun Guide):
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if applicable) or private IP of the nomad client

What’s New in Release 2.19.09

Fixes

• Better handling of non-alphanumeric characters in authentication passwords for Docker executors. Fixed a bug that was preventing the use of ~ and ? in the Docker ID password field under the auth key.
• General security updates.

Known Issues

• On static (non-AWS) installations with NONE selected under storage driver settings, tests can be split only by file name or file size. If a user attempts to split tests by timing data, static instances will split them by file name instead.
• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds (for further information on SSH reruns see the SSH Rerun Guide):
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if applicable) or private IP of the nomad client

What’s New in Release 2.19.08

Fixes

• Fixed a bug that was causing parallelism to fail for static installations with NONE selected under storage driver settings.

Known Issues

• On static (non-AWS) installations with NONE selected under storage driver settings, tests can be split only by file name or file size. If a user attempts to split tests by timing data, static instances will split them by file name instead.
• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds (for further information on SSH reruns see the SSH Rerun Guide):
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if applicable) or private IP of the nomad client

It has come to our attention that an Ubuntu 14.04 AMI in us-west-1 has become inaccessible. If you are using this AMI you should move your Nomad clients to ami-0c7b2a20526d0c5ff. For information on supplying AMIs for your installation, see the VM Service guide.

If you are still running Ubuntu 14.04 please contact us to migrate to Ubuntu 16.04.

What’s New in Release 2.19.07

Fixes

• Fixed a bug that was preventing authentication to China region ECR images in the docker executor.
• Removed the recursive chown from startup of the fileserverd service. In instances of heavy usage this was causing the startup process to take a long time, or in some cases, startup was blocked.

Known Issues

• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds (for further information on SSH reruns see the SSH Rerun Guide):
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if applicable) or private IP of the nomad client

What’s New in Release 2.19.06

Fixes

• Fixed a bug that was causing workflow statuses to be displayed on GitHub in the wrong order.
• Introduced performance improvements to workflows-conductor that dramatically reduce CPU usage and latency.
• Fixed a bug that was preventing use of an S3 storage region other than us-east-1 via an IAM user.
• Fixed formatting type of SMTP passwords to ensure they are masked during setup.

Known Issues

• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds (for further information on SSH reruns see the SSH Rerun Guide):
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if aplicable) or private IP of the nomad client

What’s New in Release 2.19.05

Fixes

• Fixed a bug that could lead to the VM database ending up in an incorrect state in the event of a Services Machine crash or manual termination of VM Service instances.

Known Issues

• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds:
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if aplicable) or private IP of the nomad client

What’s New in Release 2.19.04

Fixes

• Fixed a bug that was leading to support bundle creation timing out before the Services machine logs had been created, leaving only Replicated logs.

• S3 connection pool metrics under circle.s3.connection_pool.* have been added to the test results service, making it easier to debug issues with this service.

• Add in missing environment variables for the workflows service. The absence of these environment variables was causing excessive stack tracing whenever workflows were run. This in turn lead to excessive log rotation.

• Fixed a bug that was causing failure to update GitHub statuses. Some customers experienced this bug when a project had a follower with a broken auth token.

Known Issues

• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds:
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if aplicable) or private IP of the nomad client

What’s New in Release 2.19.03

Before Upgrading

If you are using an IAM role scoped to a non-root path, you will need to unset the OUTPUT_PROCESSOR_USE_NAIVE_ROLE_MAPPING environment variable in your output processor customization script. See the Customizations Guide in our documentation for more information on using customization scripts.

Fixes

• Removed the use of the depecated GitHub.com API endpoint GET applications/%s/tokens/%s.

• Distributed tracing is now enabled by default for Server installations. Traces are used in CircleCI support bundles to improve our ability to troubleshoot Server issues. Options for the tracing sampling rate are displayed in the Replicated Management Console, but should only be changed from the default if requested by CircleCI Support.

• Fixed an issue that was preventing restore_cache from working with the storage driver set to “none” - i.e not S3.

• Fixed an issue that was preventing the output_processor service from using AWS AssumeRole when the role was located in a subfolder. This issue affected customers with security policies forcing the use of a subfolder in this case, and the symptoms included the inability to store artifacts or use timings-based test splitting.

• JVM heap size can now be changed using the JVM_HEAP_SIZE environment variable for the following services: vm-service, domain-service, permissions-service and federations-service.

Known Issues

• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds:
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if aplicable) or private IP of the nomad client

What’s New in Release 2.19.02

Fixes

• In the LDAP login flow we now use an anonymous form to POST LDAP auth state, rather than sending it as a GET parameter. Previously, when a user authenticated using LDAP, their username and password were sent in plaintext as part of a query parameter in a GET request. As requests are over HTTPS, this left usernames and passwords in request logs, etc. This issue is now fixed.

• Optimizely and Zendesk are now removed from Server release images.

• Fixed an issue in which setting CIRCLE_ADMIN_SERVER_HTTP_THREADS or CIRCLE_PUBLIC_FACING_SERVER_HTTP_THREADS too high would prevent the frontend container from starting.

• Due to changes in the GitHub API we have removed the use of ?client_id=x&client_secret=y for GitHub, and GHE versions 2.17 and later.

• Fixed an issue that was causing intermittent failures to spin up VMs with DLC in use.

• Fixed an issue that was preventing the customization of proxy settings for Docker containers. See the Nomad Client Proxy and Service Configuration Overrides guides for more infomation.

• Fixed a bug that was preventing job steps for non-failing builds being logged when proxy settings were used for the job container.

• Removed legacy TLS versions 1.0 and 1.1, in addition, enabled 1.2 and 1.3 TLS, and specified the following ciphersuites
  • ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384
• Fixed a statsd configuration issue that meant some services were not emitting Telegraf metrics.

Known Issues

• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds:
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if aplicable) or private IP of the nomad client
• Classic Load Balancer is no longer available from this version due to the ciphersuite changes listed above. CircleCI no longer accepts requests from Classic Load Balancer, so you should move to Network Load Balancer (NLB) or Application Load Balancer (ALB).

What’s New in Release 2.19.01

Fixes

• Fixed a bug that was preventing some customers from upgrading due to a schema change in one of our library dependencies.

• Fixed a bug that was preventing some customers from inspecting builds via SSH due to a logic change in our build agent.

Known Issues

• If you are using an HTTP proxy for your installation, do not upgrade to v2.19.01, instead upgrade to at least v2.19.02. We have known issues around job step logging and docker commands customization that makes the v2.19.01 release incompatible with an HTTP proxy setup of CircleCI Server.

• If any changes have been made to your networking configuration you may need to run the following steps to ensure you can use SSH to inspect your builds:

  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if aplicable) or private IP of the nomad client

What’s New in Release 2.19.00

Requirements

• Before upgrading to Server v2.19 your Nomad launch configuration must be updated by following this guide. Server v2.19 uses Nomad version 0.9.3, so if you have externalized Nomad, contact your support engineer before upgrading.

New Features

• You can now customize resource classes for your installation to provide developers with CPU/RAM options for the Jobs they configure. For more information see our guide to customizing resource classes in Server v2.19.

• CircleCI Server installations on AWS can now be configured to work on GovCloud. Using AWS GovCloud enables you to meet a variety of regulatory requirements, such as FedRAMP, ITAR, DoD SRG, CJIS, and HIPAA. Read more about how CircleCI helps customers meet security and compliance needs here.

Upgrades and Fixes

• For v2.19 we have updated Nomad clusters from 0.5x to 0.9, reducing the accumulation of dead jobs and improving overall performance. For more information, see the Nomad changelog.

• The image used to run the RabbitMQ server has been updated to fix vulnerabilities.

• The Server-Mongo-Upgrader service has been removed from the Services machine. Server Mongo Upgrader was a migration service written to facilitate the upgrade from Mongo 3.2 to 3.6, which was done as a part of the CircleCI Server v2.15.0 release in October 2018. This means it is not possible to upgrade directly from a Server version pre v2.15.0 to v2.19.0.

Known Issues

• If you are using an HTTP proxy for your installation, do not upgrade to v2.19.x until further notice. We have known issues around job step logging and docker commands customization that make the current 2.19.x releases incompatible with an HTTP proxy setup of CircleCI Server. This issue will be fixed in a future 2.19.x patch release and the ‘safe’ version will be flagged up clearly for affected customers.

What’s New in Release 2.18.03

Features

Following our release of Windows support on the hosted offering of CircleCI, running VM-based jobs in Windows environments is now supported on CircleCI Server instances running in AWS.

The version of Windows currently supported is Windows Server 2019. Check out the Customizing VM Service Images doc for the details of how to build your own custom Windows Server 2019 image, and then follow the Server examples in the Getting Started with Windows doc for instructions on how to set up a Windows job on your CircleCI Server installation.

Deprecation

• Server-Mongo-Upgrader will be removed in Server 2.19.0. Only effects customers running < 2.15.0.

What’s New in Release 2.18.02

Bug Fix

• Fixed an error when setting CIRCLE_PUBLIC_FACING_SERVER_HTTP_THREADS or CIRCLE_ADMIN_SERVER_HTTP_THREADS above 50.

What’s New in Release 2.18.01

• Introduced configuration for timeouts for the following HTTP calls to GitHub (GHE):
  • GITHUB_CONN_TIMEOUT
    • Individual configuration for connection timeout, default 5000ms
  • GITHUB_SOCKET_TIMEOUT
    • Individual configuration for socket timeout, default 5000ms
  • GITHUB_CONN_REQUEST_TIMEOUT
    • Individual configuration for receiving connection from connection pool, default 5000ms
  • GITHUB_TIMEOUT
    • Single configuration for all of the above timeouts
• Improved performance and monitoring of output-proceessor

Bug fixes

• Fixed an issue with Nomad garbage collection that had the potential to prevent jobs from being run for heavy workloads. This issue was fixed in v2.17.1 but regressed in v2.18.

What’s New in Release 2.18

Upgrading your installation

Our upgrade guide takes you step by step through the process of upgrading to the latest version of CircleCI Server, including any checks and updates required before starting the upgrade process.

Note: If at any time your organization name has been changed, there is a script that must be run before starting the upgrade process.

Notes and Best Practices

• We now require a minimum 32GB of RAM for the Services Machine.

• We have made some changes to our Redis configuration. If you have externalized Redis then you’ll need to update your configuration. Please contact your Customer Success Manager.

• We have also made changes to our Postgres version and require at least postgreSQL v9.5.16. If you have externalized postgreSQL then please update to at least that version in 2.17.x before upgrading to 2.18.

Known Issues

Nomad garbage collection only runs every 4 hours. This could prevent nomad jobs from running in environments with heavy usage. If you hit this bug please follow this work around:

1. Create /etc/circleconfig/schedulerer/customizations on the services machine
2. Inside /etc/circleconfig/schedulerer/customizations enter the following:

   export SCHEDULERER_NOMAD_BACKPRESSURE_MAX_PENDING_JOBS=500 
   export SCHEDULERER_NOMAD_BACKPRESSURE_MAX_DEAD_JOBS=100000
   

3. Restart the picard-scheduler container:

   sudo docker restart picard-scheduler
   

Features/Improvements

• It is now possible to restrict environment variables at run time by adding security groups to contexts. Security groups are defined as GitHub teams or LDAP groups. After a security group is added to a context, only CircleCI users who are members of that security group may access or use the environment variables of the context. For more information see our guide to restricting a context.

• Customers running storage drivers external to AWS will see improved routing times when searching for build Artifacts.

• You can now customize the metrics that get output from CircleCI. For steps and options, see our Custom Metrics guide. Below is a short list of metrics that are included by default when enabling Custom Metrics:

• You can now provide individual AMIs for both Remote Docker and machine executor jobs. Previously we provided the option for a single custom AMI to be used across both, but with v2.18, this expanded customization gives you greater control over versioning and dependencies to meet your individual CICD needs. See our VM Service guide for more information.

Bug fixes

• Additional fixes around contexts and org renames.

• Fixed an issue where occasionally volumes would fail to attach to spun up Remote docker/machine instances.

• Fixed an issue where the CircleCI integration could not be installed on JIRA instances with the jira.com subdomain.

• Fixed an issue where the Workflows page would still point to an old repo after renaming an organization.

• Fixed issue where the Workflows UI would fail to refresh data automatically.

• Improved context loading times in cases when they could cause timeouts in the UI.

• Fixed an issue where contexts would cause builds to return CIRCLE_BUG.

What’s New in Release 2.17.3

New Features:

• Added ability to place banners on login screens. See our guide here.

What’s New in Release 2.17.2

New Features:

• Added and exposed the following metrics for operators:
  • picard.*
    • This will affect the execution services, the important ones are located in output-processor.
  • jvm.*
    • This will allow our operators to understand and monitor when certain services are running out of memory.

What’s New in Release 2.17.1

Bug fixes

• Addressed a configuration issue with VM Service running into rate limiting errors when calling out to the AWS API. This prevented VM Service from spinning up machine and remote_executors.
• Fixed issues around nomad garbage collection only garbage collecting every 4 hours. This could prevent nomad jobs from running in environments with heavy usage.

What’s New in Release 2.17

Known Issues

There have been reports of our backpressure settings being too aggressive due to lack of nomad garbage collecting, as well as issues with the Vm Service overwhelming AWS API Limits. If you are on 2.17, you should upgrade to 2.17.1.

Required Changes

IMPORTANT We have upgraded Replicated to version 2.34.1 which requires Docker 17.12.1. Please follow the instructions provided in the Updating Server Replicated Version guide before upgrading.

• When using Github Enterprise, operators will now need to provide CircleCI with a Github User Token in the Management Console Settings. This is required due to a change in how the Github Enterprise API allows us to verify Organizations. It is recommended that you use a shared ops account to generate this token. After upgrading to CircleCI Server v2.17, please follow these steps to enact this change:
  • Sign into your GitHub Enterprise instance.
  • Navigate to Personal Settings (top right) > Developer Settings > Personal Access Tokens.
  • Click “generate new token”. Name the token appropriately to prevent accidental deletion. Do not tick any of the checkboxes.
  • Copy the new token and paste it into the Management Console settings. (Github Integration > GitHub Enterprise Default API Token)
  • Save the settings in Replicated.

Notes and Best Practices

• Fill out this form to receive updates about CircleCI through email.
• It is currently a best practice to use a Services Machine with a minimum of 32GB of RAM. Starting in 2.18 it will become required. See docs for our recommendation(s).
• We have updated our software packages to the following versions. This is not a required update for those with externalized environments at this time, but will be when v2.18 is released.
  • Vault
    • 1.1.2
  • Mongo
    • 3.6.12-xenial
  • Redis
    • 4.0.14
• We are removing the 1.0 Single-Box options from CircleCI 2.0. We found a few critical vulnerabilities in our 1.0 build image, and we have long stopped recommending it for trials. If this is absolutely critical to your workflow please reach out to us. This does not impact people who are running 1.0 in clustered mode.

Features/Improvements

• Workflows now has a Slack Integration! Users can choose to receive Slack notifications when their workflows complete.
• Administrators can now restrict what organizations are allowed into their CircleCI installation. For more details on how to enable this feature, please see the User Management Section of the 2.17 Ops Manual.
• We changed the renamed org flow so orgs that have been renamed will no longer result in errors going forward. Existing users who had applied workarounds for this use case will now no longer require said workarounds.
• Workflows now take up less DB space and will be easier to maintain going forward.
• Improved the cache in front of GraphQL API resulting in overall improved performance.
• Added backpressure to avoid overwhelming nomad with requests, this will result in increased performance from existing nomad clusters.

• New machine executor AMIs based on Ubuntu 16.04 for AWS.

• Ubuntu 16.04 with Docker 18.06.3 has apt-daily and apt-daily-upgrade services disabled.
  • It is highly recommended that customers try to experiment with the AMIs below before officially switching:
  • The new images are as follows
    • Ap-northeast-1:ami-0e49af0659db9fc5d
    • Ap-northeast-2:ami-03e485694bc2da249
    • Ap-south-1:ami-050370e57dfc6574a
    • Ap-southeast-1:ami-0a75ff7b28897268c
    • Ap-southeast-2:ami-072b1b45245549586
    • Ca-central-1:ami-0e44086f0f518ad2d
    • Eu-central-1:ami-09cbcfe446101b4ea
    • Eu-west-1:ami-0d1cbc2cc3075510a
    • Eu-west-2:ami-0bd22dcdc30fa260b
    • Sa-east-1:ami-038596d5a4fc9893b
    • Us-east-1:ami-0843ca047684abe87
    • Us-east-2:ami-03d60a35576647f63
    • Us-west-1:ami-06f6efb13d9ccf93d
    • us-west-2:ami-0b5b8ad02f405a909
  • They are replacing:
    • Ap-northeast-1:ami-cbe000ad
    • Ap-northeast-2:ami-629b420c
    • Ap-south-1:ami-97bac2f8
    • Ap-southeast-1:ami-63b22100
    • Ap-southeast-2:ami-dd6c73be
    • Ca-central-1:ami-d02c93b4
    • Eu-central-1:ami-b243eedd
    • Eu-west-1:ami-61de3718
    • Eu-west-2:ami-904e5ff4
    • Sa-east-1:ami-c22057ae
    • Us-east-1:ami-05b6e77e
    • Us-east-2:ami-9b4161fe
    • Us-west-1:ami-efc9e08f
    • Us-west-2:ami-ce8c94b7

Bug fixes

• Fixed some bugs related to GitHub API response handling and webhook handling.
• Fixed issue with Scheduled Workflows when the services machine is restarted.
• Fixed changing the RabbitMQ hostname for Scheduled Workflows when externalizing.
• You can no longer create contexts with no names. If you are using a context with no names, it will no longer be accessible from your execution environment.
• We have optimized our handling of large amounts of build output and test results XML, to avoid out-of-memory errors.
• The CIRCLE_PULL_REQUEST environment variable was not being populated correctly in all cases when building across forks. This has been fixed.
• Fixed a bug where some commits with [ci skip] in the message were still being built.
• Fixed a bug causing Workflows to get stuck when infrastructure_failure happens after a job fails.
• Fixed a bug causing duplicate docker networks on same nomad client (if running build using machine:true AND vm-provider=on_host).
• Improved performance when using local storage. Previously, caching issues had been experienced when local storage was used rather than the default option of using S3 (selecting None under Storage Driver options from the Management Console).
• We have added more error checking and validation around Github’s API so the existing list commit endpoint no longer causes issues.
• Datadog API token field was stored in plaintext, now set as a password field.
• Fixed issue where workflows were constrained from fanning out to large number of jobs.

Fill out this form to receive updates about CircleCI through email.

What’s Fixed in Release 2.16.2

• Bug Fix: Fix a race condition that causes some threads to be stuck in certain use cases.

Fill out this form to receive updates about CircleCI through email.

What’s Fixed in Release 2.16.1

• Bug Fix: This release includes a new flag to disable GitHub cache warming. Note: Only add this flag if a high thread count is impacting the API service. To disable cache warming after completing the upgrade, complete the following steps:

1. Add the export DOMAIN_SERVICE_REFRESH_USERS=false flag to the /etc/circleconfig/api-service/customizations file on the Services machine.

2. Restart CircleCI by going to the Administrative console and clicking Stop Now, waiting for it to stop, then clicking Start.

• Bug Fix: Fixed issue preventing metrics from domain service from being forwarded.

• Security Fix: The SMTP password field in the dashboard was set to plain text. The password field will now be hidden.

Fill out this form to receive updates about CircleCI through email.

Note: It is currently a best practice to use a Services Machine with a minimum of 32GB of RAM. Starting in 2019 it will become required. See docs for our recommendation(s).

What’s New in Release 2.16

The following updates and fixes are also of note:

IMPORTANT We have upgraded Replicated to version 2.29.0 which requires Docker 17.12.1. Please follow the instructions provided in the Updating Server Replicated Version guide before upgrading.

New Feature We are excited to announce that you can now distribute your data and workload external to the Services Machine. The following services can be externalized; MongoDB, Redis, Nomad Server, RabbitMQ, Postgres and Vault. Please contact your CSM for the latest documentation.

New Feature Custom Metrics can now be accomplished via a Telegraf Output Configuration File. See the Custom Metrics section of the Monitoring section. Note: The Custom Metrics section in the admin UI been disabled in favor of a configuration file.

New Feature Users can now receive email notifications about Workflows

New Feature The PostgreSQL image has been updated to allow modifying the default configuration by creating the following file: /etc/circleconfig/postgres/extra.conf A list of configuration options can be found here.

Updated Removing EOL Banner on build emails

Updated VM Service stability improvements

Bug Fix Fix issue with contexts that would break after 32 days

Bug Fix Vault auth tokens will now renew periodically to prevent expiration

Bug Fix Fix for an issue where workflow listing pages would intermittently fail to render

Bug Fix We have fixed a small number of bugs affecting processing incoming web hooks from GitHub to CircleCI.

Bug Fix For installs that are using an HTTP or HTTPS proxy, the jobs will now ignore that proxy so that setup_remote_docker works.

Bug Fix Reduced the number of connections 1.0 builders make to the PostgreSQL database.

Security Fix Security fixes for potential cross-site scripting vulnerability and HTTP header injection vulnerability

Security Fix Forked PRs can no longer write the caches of parent projects by default for security reasons. It is possible to still write parent project caches from the fork if the “Pass secrets to builds from forked pull requests” (in Advanced settings) is enabled.

Update Improved metrics for VM provisioning for machine executor. This changed metric names, so if you are monitoring VM provisioning in your install already, you will need to reconfigure the monitoring dashboards for these new metrics.

• New metrics:
  • vm-service.gauges.available-vms and vm-service.gauges.running-vms replaced by vm-service.gauges.vms_by_status
  • vm-service.gauges.running-tasks replaced by vm-service.gauges.tasks_by_status
  • vm-service.gauges.oldest-unassigned-task replaced by vm-service.gauges.unassigned_tasks_age

Known Issue(s)

When a saved context is removed during the build graph phase it can cause a silent failure during job execution.
In some instances the metrics section becomes hidden in the settings page. Please see this Knowledge article for instructions.

Fill out this form to receive updates about CircleCI through email.

Note: It is currently a best practice to use a Services Machine with a minimum of 32GB of RAM. Starting in 2019 it will become required. See docs for our recommendation(s).

What’s New in Release 2.15

This release contains an upgrade to MongoDB, service migrations for contexts and is a required upgrade. The following updates and fixes are also of note:

MongoDB Upgrade The MongoDB version running on Services VM will be automatically updated from 3.2 to 3.6 for you. If you have externalized MongoDB and wish to upgrade to 3.6, please contact your CSM

Bug Fix Restrict interaction of Workflows ‘Rerun’ dropdown to users who are part of GitHub organization for open-source projects.

Bug Fix Known issue for broken UI due to workflows with “.” in the name

Bug Fix Known performance issue with freezed browsers for organizations with many branches.

Bug Fix Known issue for Vault auth tokens not being automatically renewed. If you have externalized Vault, please contact your CSM before you upgrade to 2.15.0

New Feature: Splitting test files via Workflows. If your project has a large number of tests, you can use circleci tests split feature to reduce your job run time. For more information on how to setup tests split, please follow this document.

New Feature: Workflows UI will now show actors who have taken the following actions:

• Rerun workflows
• Approved jobs
• Canceled jobs

New Feature We are excited to announce more Metrics! Nomad job metrics can be enabled and will be emitted by the Nomad Server. Please see Nomad Metrics documentation for instructions on how to configure Nomad Clients. The following five types of metrics are reported:

• circle.nomad.server_agent.poll_failure: Returns 1 if the last poll of the Nomad agent failed, otherwise it returns 0.
• circle.nomad.server_agent.jobs.pending: Returns the total number of pending jobs across the cluster.
• circle.nomad.server_agent.jobs.running: Returns the total number of running jobs across the cluster.
• circle.nomad.server_agent.jobs.complete: Returns the total number of complete jobs across the cluster.
• circle.nomad.server_agent.jobs.dead: Returns the total number of dead jobs across the cluster.

New Feature Host (Services VM) and docker metrics can now be forwarded via Telegraf , a plugin-driven server agent for collecting & reporting metrics, to DataDog or a custom location. To enable and/or configure please go tohttps://example.com/settings#metrics. For more information please see the Monitoring documentation.

Performance Improvement Reduced the load time for all workflows pages.

Performance Improvement CircleCI has upgraded the mechanism that returns gettimeofday calls. As a result, heavy users of syscall will see an aggregate improvement in performance. Most noticeably this will improve the run time of projects with heavy usage of debugging/profiling tools and Xdebug with PHP.

Updated JVM_HEAP_SIZE is configurable for frontend, output-processor, and test-result containers. Please see JVM Heap Size Configuration documentation.

Updated For customers running the Static installation, Ubuntu 16.04 (Xenial) has been added as a supported OS. For additional details please see the docs.

Known Issue(s)

When a saved context is removed during the build graph phase it can cause a silent failure during job execution.
When using the none storage driver, the cache will not be saved. This means that you currently cannot use caching in CircleCI when you are using the none storage driver.

This patch release fixes an issue introduced in 2.14.0. Please review the fix(es):

Fixed: Docker executor does not support configurable resource classes.

What’s New in Release 2.14

This release contains service migrations for contexts and is a required upgrade. The following updates and fixes are also of note:

Fixed: Pre-allocated VMs not being used. Customers reported that jobs using machine executor and setup_remote_docker were continuing to spin up (and terminate) new VMs at runtime instead of the pre-allocated pool.

Fixed: Unable to build in AWS region eu-west-2.

Updated: The title of the Builds page in the CircleCI application was changed to Jobs in several places to better reflect the structure of builds.

Updated: Contexts are now checked for unique naming across your installation of CircleCI instead of just within an organization. You will no longer be able to create duplicate context names within an account. Existing duplicates will be honored by the system, but should be changed to avoid future collisions. Please see the Contexts documentation for further information.

Updated: The test-results services cleans up the /tmp directory on restart and the JVM heap size increased from 256m to 512m.

Known Issue(s)

When a saved context is removed during the build graph phase it can cause a silent failure during job execution.
When using the none storage driver, the cache will not be saved. This means that you currently cannot use caching in CircleCI when you are using the none storage driver.

Migration Tools: The 1.0 migration center was updated with additional content and helpful tools to migrate projects from CircleCI 1.0 to 2.0.

What’s New in Release 2.13*

• Fixed: Due to a certificate not being imported jobs were unable to submit after a container restart.
• Fixed: Stability issues within contexts service.
• Fixed: vm-service issue where AWS ECS and EBS volumes were left used. Added better idempotent client retries, API eventual consistency handling.
• Added: vm-service EBS Volumes cache evictor. Docker Layer Caching EBS volumes get deleted after 14 days of unused. You can control the inactivity period in Installation Admin Settings page.
• Added: Periodic checks to scale down any unused long running instances (e.g. due to reduction in prescaling configurations, or or due to a bug preventing termination of VMs in timely manner).
• Added: Optimizations to speed up provisioning. Resiliency to network/external changes.
• CircleCI is excited to announce that users on AWS can now forward host (Services VM) and docker metrics via Telegraf , a plugin-driven server agent for collecting & reporting metrics, to AWS CloudWatch. This is is the first step into bringing you greater observability into your installation. To enable and configure please go tohttps://example.com/settings#metrics. In this release the following metrics may be enabled: Docker , Networking , Disk , Memory, and CPU. For more information please see the Monitoring documentation.
• CircleCI is enhancing visibility by capturing usage statistics of CircleCI installations. This aggregate data will be used to better support you and ensure a smooth transition off 1.0. This optional feature is automatically available when you upgrade to the latest version of your installation or once a support bundle is downloaded. Below are the metrics being captured. If you would like access to this data in order to better understand your users please contact a CircleCI account representative. Please reference exhibit C within your terms of contract and our standard license agreement for our complete security and privacy disclosures. For more information please see the Usage Statistics documentation.

Weekly Account Usage

Weekly Job Activity

* this release contains migrations and is a required upgrade

The installable version of CircleCI v2.12.1 fixes an issue introduced in 2.12.0. GitHub cloud integrations were unable to create new contexts.

The installable version of CircleCI v2.12.0 fixes an issue in which an error occurs when uploading artifacts to S3 by updating AWS SDK for Java from v1.10.69 to v1.11.328.

This release fixes an issue with overwriting the known_hosts.local file. The fix allows keys to remain and enables the CircleCI upgrade process to append keys.

The installable version of CircleCI v2.10.0 fixes an issue in which jobs were unable to submit after a container restart due to the certificate not being imported.

The installable version of CircleCI Server 2.9.0 includes the following improvements and bug fixes:

• Resolution for a potential LDAP injection vulnerability
• Addition of support for the following LDAP fields:
  • Group Membership
  • Group Object Classes

See the Authentication document for the steps.

• Addition of default values for OpenLDAP and Active Directory
• Stability and bug fixes for an issue with test results
• Improvements to ready agent logging for support bundles
• Stability fixes and speed improvements for provisioning remote Docker engine
• Reliability improvements for the permissions service client
• Differentiation between the LDAP login and email to remove email as the assumed login value
• Fix for a bug that deselected all projects when a new user logged in for the first time
• Fix for a bug that was preventing Workflows from working correctly when located behind a proxy
• Bug fixes and improvements for Contexts

The installable version of CircleCI 2.8.0 includes the following fixes and improvements:

• An issue involving pre-allocated remote docker, machine VM creation, and other VM usage on AWS when the on-host VM option is selected has been fixed.

• An issue with the VM provider timing out when waiting for SSH to connect has been fixed. This upgrade supersedes any custom values for the Number of SSH attempts for starting VM setting and adds an equivalent field named Timeout for waiting for SSH to connect to VM(s) that you must set.

The installable version of CircleCI 2.7.0 includes the following fixes and improvements:

• Improvements to remote Docker instance provisioning and cleanup on AWS.

The installable version of CircleCI 2.6.0 includes the following fixes and improvements:

• It is now possible to create multiple Contexts in the Organization Settings of the CircleCI app. See the Contexts documentation for details and naming conventions.

The installable version of CircleCI 2.5.0 includes the following fixes and improvements:

• This release fixes a problem with DNS resolution failure on alpine-based containers. This problem was observed in the 2.3 release and was fixed for some service containers in 2.4. The 2.5 update fixes the issue for all service containers.
• This release includes improvements to scaling and garbage collection in the VM Service that should improve performance for installations with heavy use of machine executors or remote Docker.

The installable version of CircleCI 2.4.0 includes the following improvements and changes:

• Audit logging - This feature makes available a downloadable log of system events from the CircleCI Admin dashboard. This is the first release of audit logging; more events and improved data in the logs will be delivered in subsequent releases. Find detailed information about the data fields for each event in the Audit Logging section of the Security Features document.

• LDAP integration - It is possible to enable a new CircleCI installation to authenticate users with OpenLDAP or Active Directory credentials. New installations may be configured to have users authenticate through LDAP, including Active Directory support. Refer to the Authentication document for instructions.

• New build-agent - The new build-agent fixes some cases in which users were unable to log into a running job with SSH.

• VM Service improvements - Autoscaling of the VM Service has been improved, so machine executors should have more consistent performance at scale.

The installable version of CircleCI 2.3.0 includes the following improvements and changes:

• Users can now schedule workflows. The underlying cron-service has been added to the installation, though no specific configuration should be needed. See the documentation on Workflows Orchestration for details.
• Support for using private IP addresses in 1.0 builds is now enabled along with 2.0 builds
• A bug in the VM garbage collector was addressed that should help in situations when VMs were being leaked.
• We have disabled Replicated snapshots. Please see the documentation on backups if you have previously used Replicated snapshots.

The installable version of CircleCI 2.2.0 includes the following improvements and changes:

• New visualization for Workflows - When you click on a workflow in the CircleCI application, a diagram appears with the relationship of your jobs to one another and the complete sequence of the workflow.
• Updates to the onboarding flow - We now show a new user only projects already building on CircleCI rather than a list of all projects found in source control. This change was made to prevent inadvertent adding of projects to CircleCI during the onboarding process. We also added a confirmation dialog for any user opting to follow more than six projects and made it clearer how to add new projects not yet building on CircleCI.
• Support for object stores other than S3 - CircleCI 2.2.0 now supports the same object store options that are supported in 1.x. This is a prerequisite to future releases that will allow non-AWS installations of CircleCI 2.x.
• Updates to the VM Service; ACTION MAY BE REQUIRED IF YOU ARE USING AWS - The VM service has been updated to use volumes rather than persistent instances. This change allows operators to accommodate Docker Layer Caching with a smaller number of long-running VMs. If you installed using the suggested IAM permissions from prior versions you must update your IAM permissions with this upgrade. Refer to the policy template for the complete set of permissions.
• New Contexts feature; ACTION MAY BE REQUIRED IF YOU ARE USING THE HIGH AVAILABILITY CONFIGURATION - The Contexts feature uses a HashiCorp Vault secure storage mechanism. If your installation is running in HA mode, HashiCorp Vault needs to be externalized similar to the Mongo and Postgres databases. Refer to the Configuring High Availability documentation for instructions.

• Fix setting for certain proxy configurations.

• Added support for 2.0 builders to use object storage other than S3 (mirroring the options from the 1.x releases).
• Support for private images in docker jobs, including support for ECR authentication.
• New VM service and build agent to address certain situations that resulted in build infrastructure failures.
• Better accommodations for using proxies throughout the core infrastructure.

• Added support for proxies to the VM Service/Remote Docker setup.
• Fixed an issue in the mongo migration scripts when moving from 1.x to 2.x that caused some customers problems.
• Move to using Docker 17.03.2 (downgrade) to solve occasional freezes when starting/restarting.

• Fix an issue where users were unable to import large PostgreSQL data sets during an upgrade from 1.0 to 2.0

• Fix issue where /etc/hosts was not being applied properly to every container.
• Add ability to configure VM instance type in AWS EC2 when configuring the VM Service in the Management Console.

CircleCI 2.0 is a completely updated CI/CD platform that starts every run with a clean image which is automatically provisioned just-in-time for Linux jobs on the CircleCI application installed in your private cloud. The CircleCI 2.0 platform includes significant performance, stability, and reliability improvements along with first-class Docker support, Workflows (pipelines), Resource Allocation, and Contexts. The Management Console (Replicated) also includes new options for enabling 2.0 Builders and appropriate default settings.

If you have CircleCI installed you may access CircleCI 2.0 features on your current installation with no restrictions under your current agreement and support level. However, you must contact your CircleCI account representative for assistance with upgrading, to obtain migration scripts, and to receive a new license channel.

• CircleCI 2.0.0 is only available on AWS and may be installed with Terraform or manually.
• Teams may create a CircleCI 2.0 .circleci/config.yml file in their repositories to add new 2.0 projects incrementally while continuing to build 1.0 projects which use a circle.yml configuration file.

Documentation for the installable 2.0 application is available at CircleCI 2.0 Documentation for the following topics:

• Administrator’s Overview
• Administration FAQ
• Security Features
• Installing CircleCI 2.0 on Amazon Web Services with Terraform
• Configuring High Availability
• Administrative Variables, Monitoring, and Logging
• Introduction to Nomad Cluster Operation
• Backing up CircleCI Data

• Addresses an issue with CloudWatch metrics not being reported correctly.
• Address an issue with some build output being truncated in the UI.

• Addresses an issue with CloudWatch metrics not being reported correctly. UPDATE: some customers report this has not addressed their CloudWatch problems. The next release will have a new fix.

• Addresses an issue for installations pointing at github.com (rather than an instance of GitHub Enterprise) that prevented those installations from adding new projects.

• Fixes a bug that was causing lists of builds to be truncated in some circumstances.
• Fixes a bug where certain configurations of GitHub were not syncing email addresses properly.
• Fixes a bug where some users were seeing a beta UI that should have not have been showing up.
• Various minor UI fixes.

• Fixes a bug in new installs that prevented the first user from becoming an Admin.
• Fixes a bug with using the built-in Heroku deployment support.

• New Projects list in the Admin section gives you an overview of all projects across your entire installation of CircleCI Enterprise.
• New onboarding flow – new users will be prompted with a list of projects to follow drawn directly from their GitHub account.
• New option for users to choose “My builds” or “All builds” when viewing lists of builds.
• New JIRA integration for creating issues from a Build Page - configured under Project Settings.
• Option to externalize the underlying persistance storage (Postgres and Mongo) - talk to your CircleCI contact for details.
• Improved support for snapshot backup and restore via the Admin Console.
• Various performance improvements and minor bug fixes.

This is a security update to prevent logging temporary access tokens in build output. Those access tokens would have allowed a user to do a PUT request to overwrite an existing artifact only if all of the following were true:

1. They already had access to the build.
2. They identified the token in the build output.
3. The were to use the token within 30 minutes after the original artifact was uploaded.

A minor maintenance release, primarily to improve the single-box installation process for new trials of CircleCI Enterprise. Also, we have now follow advice from GitHub to not check if emails in GitHub Enterprise are “verified” — we will continue to check verified status if you are using github.com. If you have reason to change “verified” checks on your users’ emails let us know.

Note for Admins

This release requires extra steps than usual. Please see <(https://circleci.com/docs/enterprise/upgrade-for-1.46.x-and-earlier/)> for detailed instructions.

We have released patches to our AMIs and other infrastructure to address CVE-2016-8655. We recommend all CircleCI Enterprise installations follow the instructions below to update both their Services box and their Builder fleet.

If you have any questions or difficulties please contact enterprise-support@circleci.com.

Update the Services box:

1. As always, ensure your data is backed up.
2. Shut down CircleCI in the Replicated console (or via the CLI).
3. Update the kernel using the provided install_kernel_master function below.
4. Restart the machine.

#!/bin/bash
function install_kernel_master() {
    echo '>>> Installing Kernel'
    apt-get update
    apt-get install linux-image-3.13.0-105-generic linux-headers-3.13.0-105-generic linux-image-extra-3.13.0-105-generic
    apt-cache policy linux-image-3.13.0-105-generic linux-headers-3.13.0-105-generic linux-image-extra-3.13.0-105-generic
}

Update the Builder fleet:

For the builder fleet, update the Launch Configuration to use the updated AMI from the list below:

• ap-northeast-1 = “ami-07f09d60”
• ap-northeast-2 = “ami-90588ffe”
• ap-southeast-1 = “ami-4f54f82c”
• ap-southeast-2 = “ami-040c3467”
• eu-central-1 = “ami-3465a35b”
• eu-west-1 = “ami-d70421a4”
• sa-east-1 = “ami-2e6bf242”
• us-east-1 = “ami-e68f89f1”
• us-west-1 = “ami-901c4af0”
• us-west-2 = “ami-0c57fc6c”

If you are using our Terraform scripts, you can download the new script https://github.com/circleci/enterprise-setup/blob/master/circleci.tf and run terraform apply. We’ve already updated the scripts to include the new AMIs, so terraform should launch new builders automatically with the patched version, and cycle your fleet.

If you are using a non-AWS environment, use the same method to patch your builders you used to patch the Services box.

If any of the above does not apply to your environment, or you encounter issues with your upgrades please contact: enterprise-support@circleci.com.

• Fix for a bug causing builders to not honor CIRCLE_CONTAINERS_SUBNET when passed in the initialization.
• Builds with high parallelism were experiencing slow performance on the build detail page in the web UI – this should now be improved.
• Some emails generated by CircleCI had hard-coded links to circleci.com. These should now point at your installation host.
• A handful of minor UI fixes.

We have released patches to our AMIs and other infrastructure to address CVE-2016-5195. We recommend all CircleCI Enterprise installations follow the instructions below to update both their Services box and their Builder fleet.

If you have any questions or difficulties please contact enterprise-support@circleci.com.

Update the Services box:

1. As always, ensure your data is backed up.
2. Shut down CircleCI in the Replicated console (or via the CLI).
3. Update the kernel using the provided install_kernel_master function below.
4. Restart the machine.

#!/bin/bash
function install_kernel_master() {
    echo '>>> Installing Kernel'
    apt-get update
    apt-get install linux-image-3.13.0-100-generic linux-headers-3.13.0-100-generic linux-image-extra-3.13.0-100-generic
    apt-cache policy linux-image-3.13.0-100-generic linux-headers-3.13.0-100-generic linux-image-extra-3.13.0-100-generic
}

Update the Builder fleet:

For the builder fleet, update the Launch Configuration to use the updated AMI from the list below:

• ap-northeast-1 = “ami-59389e38”
• ap-northeast-2 = “ami-a2e236cc”
• ap-southeast-1 = “ami-a80fa9cb”
• ap-southeast-2 = “ami-53241930”
• eu-central-1 = “ami-5a59a035”
• eu-west-1 = “ami-e0c78993”
• sa-east-1 = “ami-3832af54”
• us-east-1 = “ami-d0396bc7”
• us-west-1 = “ami-76226916”
• us-west-2 = “ami-938420f3”

NOTE: The following AMIs are not maintained by CircleCI, but contain a patched version of stock ubuntu.

• gov-west-1 = “ami-34df6755”
• cn-north-1 = “ami-92f622ff”

If you are using our Terraform scripts, you can download the new script https://github.com/circleci/enterprise-setup/blob/master/circleci.tf and run terraform apply. We’ve already updated the scripts to include the new AMIs, so terraform should launch new builders automatically with the patched version, and cycle your fleet.

If you are using a non-AWS environment, use the same method to patch your builders you used to patch the Services box.

If any of the above does not apply to your environment, or you encounter issues with your upgrades please contact: enterprise-support@circleci.com.

• Fix for a bug that stopped certain CloudWatch metrics from working.

• Fix for a bug that caused UI problems for certain Pull Requests

• Minor changes to help debug some esoteric issues.

• Fixed layout problems in Safari on Mac.
• Fixed issue with tag builds interacting with the Auto-Cancel of Redundant Builds setting.
• Fixed performance of the build page, including awkward interaction when expanding build steps
• Fixed environment variables not sorting properly in project settings

• Note for Admins: we will now block your upgrade if there are pending migrations that need to be performed against your databases.
• We have changed the default behavior for builds with parallelism set above the current total container count in your build fleet. Previously we would auto-cancel any build with a parallelism above the total number of containers in your fleet. Those builds will now remain in the queue until the fleet is big enough to accommodate their parallelism setting. Builds behind those builds will continue to be bumped ahead, however, so the behavior is somewhat different than normal queuing behaviors, which is a strict FIFO model.
• New System Settings option to set the maximum parallelism for all projects. This allows you to prevent teams from using more of your fleet than intended.
• New option to auto-cancel “redundant” builds on a branch - found in the project settings, this new option allows you to set builds on branches other than your default branch to auto-cancel any builds on that same branch already running. This avoids running multiple simultaneous builds on the same working branch if you push several times in quick succession.
• On the Systems Settings page under Admin you can now enter “Container Tweaks” that will be run whenever containers are started in your build fleet. This feature allows you to make adjustments to the execution environment prior to builds running, so you can save time in each build. Note that these tweaks are only appropriate for things you want to apply to the environment for ALL of your builds.
• We now parse your circle.yml before the build runs and fail if there are errors. Before, the build would start and errors would be surfaced only once they caused a failure in a running build.
• Fixed a bug preventing artifacts from being generated in some circumstances after a failed build.
• The Artifacts API now allows cross-origin requests when requesting specific artifacts. In the past you could do this when using the API to get a list of your artifacts but not for downloading specific artifacts. You can now do both with the API.
• Fixed a bug that prevented some cases of triggering builds via the API. If you have had problems triggering builds with the API you may need to recycle your build fleet for this fix to take effect.
• Fixed a bug that would cause problems if someone’s access was removed from a project in GitHub Enterprise but that person’s token was being used by CircleCI Enterprise. Other users should now be able to continue using the project as expected if the original token’s owner no longer has access.
• Fixed a bug that prevented builds cancelled from the UI from actually cancelling in a timely fashion.
• Various design improvements such as better notifications and confirmation dialogs when changing various settings, new links on the build page and builds list to make it easier to link directly to specific containers or parts of your build output, better information for builds marked as “Not Run”, hiding some settings that aren’t applicable to Enterprise customers, and various minor improvements and bug fixes.

• Fix for an issue that was causing builds to fail that were triggered by users with a github account but no circleci account.

Note: we are shifting our versioning to better reflect the current release semantically.

• PR-only builds - We have added functionality to only run builds when a pull request is open. To enable this functionality you can navigate to “Advanced Settings” for your project and enable the “Only build pull requests” option.
• Chrome notifications - see the docs
• Fixed a bug where builds triggered by people who don’t have seats in CircleCI Enterprise were not running. They now should run as before.
• Some UI changes to make updating data in the web interface more consistent.
• Reliability improvements by changing retry logic in various places.

Bug fix: We’ve fixed a bug that “–branch” option was missing when “–single-branch” is used.

New Feature: The maximum size of files that you can upload for caching during builds has been fixed to 5G. The size is now bumped 20G by default on CircleCI Enterprise and customers can also override the default value to even larger size.

To override the default value, you need to run the following in a REPL of Service Box.

(settings/set {:max-file-cache-bytes <max-size-by-bytes>})

To unset the overridden value and use default, run the following.

(settings/unset! [:max-file-cache-bytes])

New Feature: Custom base URL for version control webhooks. When a new project is added, CircleCI will add a webhook to the GitHub repository of the project. With this new feature, you can override the default webhook base URL from the System Settings page under the Admin tools (available to designated administrative users). This feature is useful when your instance is behind firewall or other proxy and cannot directly receive webhooks from GitHub.

This release fixes a bug in the URL structure used to serve build artifacts.

The artifact URL format was recently changed to handle security concerns related to CircleCI’s hosted offering. The security concerns do not affect CircleCI Enterprise customers, but the change caused issues fetching build artifacts in CircleCI Enterprise installations. This release reverts the CircleCI Enterprise artifact format and resolves the issue.

• Please Note: If you are using OS X builds you will need to run a manual migration as part of this release. After upgrading you will need to run this in a REPL: (circle.backend.model.esxi-vm/run-migrations!). Please talk with your account manager if you need further instructions.
• The Admin Users page now has links to the builds of each user.
• The API has a new endpoint /api/v1/admin/licensing that returns information about the number of seats available, number being used, when the license will expire, and how many inactive users there are in the system.
• Fixed a bug where links to the documentation broke.
• Various small bug fixes and UI improvements.

Please make sure that you are using the latest provision-builder.sh and init-builder-0.2.sh before you upgrade to this release:

https://s3.amazonaws.com/circleci-enterprise/provision-builder.sh

https://s3.amazonaws.com/circleci-enterprise/init-builder-0.2.sh

As part of this release we’re changing the behavior of artifacts to only serve an allowed set of content-types. This means we won’t serve .html files as text/html. This is a security risk on CircleCI Enterprise since artifacts are served on the same domain as the rest of the site – as a result, any user or malicious code used as part of your build can push a specially-crafted artifact and gain control of another user’s account.

If this is an issue, you can override this behavior by setting “Serve artifacts with unsafe content-types” in the admin console. We don’t recommend this, but we’re providing it for backwards compatibility.

This release also includes some changes to container networking. Containers now each use a /24 in the subnet 172.16.1.0/16 by default.

If this conflicts with your private network, or if you were editing lxc-net manually in order to fix a prior conflict, you can now use CIRCLE_CONTAINERS_SUBNET and CIRCLE_CONTAINERS_SUBNET_NETMASK_LENGTH on the builders to configure those.