CircleCI Server v3.x What’s New

Last updated
Tags Server v3.x Server Admin

Server 3.x is now generally available. The newest version of server offers the ability to scale under heavy workloads, all within your own Kubernetes cluster and private network, while still enjoying the full CircleCI cloud experience.

Server 3.x includes the latest CircleCI features, such as orbs, scheduled workflows, matrix jobs, and more. For existing customers interested in migrating from 2.19 to 3.x, contact your customer success manager. Server 3.x will receive monthly patch releases and quarterly feature releases.

Release 3.2.2

Upgrade notes

  • The rerun workflow endpoint now returns workflow ID rather than the message accepted.

Fixes

  • TLS is terminated outside of frontend so the SSL server has been completely removed from the frontend container.

  • Moved the default certificate logic from KOTS to helm.

  • Fixed the build agent image version used in server v3.x. The image used previously was causing problems with runner.

Known issues

  • Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised, for example, by using a VPN into your VPC.

  • It is currently possible for multiple organizations under the same CircleCI server account to have contexts with identical names. This should be avoided as doing so could lead to errors and unexpected behavior.

  • CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

  • The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.

  • Runner cannot be used when server is installed behind a proxy.

  • Let’s Encrypt certificate generation does not work. You will need to provide your own certificates or use the default certificates provided.

Release 3.2.1

Upgrade notes

From the KOTS admin console dashboard, select Version History from the menu bar and click Deploy for server v3.2.0.

See Upgrade notes before upgrading from v3.1.x to v3.2.x.

New features

  • Private VMs are now supported for installations on GCP.

Fixes

  • mTLS is now disabled by default.

  • SSH timeout for VMs has been increased to 10 minutes.

  • Private VMs now request private IPs.

Known issues

  • Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised, for example, by using a VPN into your VPC.

  • It is currently possible for multiple organizations under the same CircleCI server account to have contexts with identical names. This should be avoided as doing so could lead to errors and unexpected behavior.

  • CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

  • The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.

  • Runner cannot be used when server is installed behind a proxy.

  • Let’s Encrypt certificate generation does not work. You will need to provide your own certificates or use the default certificates provided.

Release 3.2.0

Upgrade notes

From the KOTS admin console dashboard, select Version History from the menu bar and click Deploy for server v3.2.0.

When upgrading from server 3.1.x to 3.2 there will be some downtime due to a change to the PostgreSQL pod. There are two issues you could run into with this update, which are covered in the following sections.

PostgreSQL pod stuck in pending

If you find that the PostgreSQL pod is stuck in a pending state after upgrading, scale down the pods to 0 and then scale up again by following the steps below.

To check if your postgreSQL pod is stuck in pending:

$ kubectl get pod -l app.kubernetes.io/name=postgresql
NAME           READY   STATUS    RESTARTS   AGE
postgresql-0   1/1     Pending   0          3m

The following command will scale down pods to 0 and terminate the application pods without any data loss:

kubectl scale deployment -l layer=application --replicas 0

Once all the application-layer pods have finished terminating do one of the following

  • either redeploy the update from the KOTS admin console

  • or run the following two commands to redeploy the pods and return server to a functional state:

    kubectl scale deployment -l layer=application --replicas 1

    Then scale output-processor up with the following command:

    kubectl scale deployment output-processor --replicas 2

Traefik pod fails to schedule

If you find that there are two Traefik pods after upgrading, you will need to locate the older pod and remove it to allow the new pod to schedule correctly.

To see the status of your Traefik pod:

$ kubectl get pod -l app=traefik
NAME                                      READY   STATUS    RESTARTS   AGE
circleci-server-traefik-9d6b86fd8-f7n2x   1/1     Running   0          24d
circleci-server-traefik-cf7d4d7f6-6mb5g   1/1     Error     0          3m

Remove the older Traefik pod with the following command:

kubectl delete pod circleci-server-traefik-<older pod hash>

The new Traefik pos will then start ot schedule correctly.

New features

  • Customers who require a fully private installation can now access a setting in the KOTS admin console to ensure public IPs are not assigned to VMs. Note that with this non-public IP setting enabled, a work-around will be needed if SSH access to running jobs is required, for example, by using a VPN into your VPC.

  • Customers that manage outbound traffic through a proxy can now configure proxy settings through the KOTS admin console. Please see our documentation for specifics on proxy support for server.

  • We have expanded the machine execution environment options available to include additional resource classes, sizes, and executors. You now have access to Arm (medium, large), Linux (medium, large, X large, and XX large), and Windows (medium, large, XX large) resource classes.

  • The insights API is now available to all server customers. Leverage build and other data to better understand the performance of teams and the health of your build and testing efforts.

  • We have revamped the admin UI, and updated our installation instructions, making it easier to set up and manage server.

  • You can now supply a custom Linux AMI for VM service.

  • SSL termination can now be disabled - If you have put server login behind a firewall, this will enable SSL termination at the firewall.

  • You can now control the size of persistent volumes. For larger customers, the initial persistent volume size was too small, by default. You can now set this at install time, providing an easier migration for those customers that require it. For further information see the Internal Database Volume Expansion doc.

  • We have added an auto-scaling example to the nomad client terraform.

  • You can now choose to serve 'unsafe' build artifacts. Previously this option was hidden, meaning potentially unsafe artifacts were rendered as plain text. For more information see the Build Artifacts doc.

Fixes

  • The default windows executor was not as documented, we have increased the size to align with documentation and cloud.

Known issues

  • KOTS admin configuration incorrectly selects the Nomad mTLS as enabled during setup. It should be set to mTLS disabled until after nomad clients have been deployed.

  • Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for private installations, but for public installs you would need to ensure that you can access the private IP advertised, for example, by using a VPN into your VPC.

  • It is currently possible for multiple organizations under the same CircleCI server account to have contexts with identical names. This should be avoided as doing so could lead to errors and unexpected behavior.

  • CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

  • The KOTS admin console cannot be upgraded if your installation is set up to be behind a proxy. The proxy settings will be deleted and cause the KOTS admin console to break.

  • Runner cannot be used when server is installed behind a proxy.

  • Let’s Encrypt certificate generation does not work. You will need to provide your own certificates or use the default certificates provided.

Release 3.1.0

Upgrade notes

With this release, the frontend-external load balancer has been removed. The traefik load balancer now handles all incoming traffic. When updating from a previous server 3.x version, you will need to update the DNS record that was pointing to the frontend-external load balancer and have it point to the circleci-server-traefik load balancer instead. Remember, you can retrieve the external IP address or DNS name of your traefik load balancer by typing kubectl get svc/circleci-server-traefik in a terminal that has access to the cluster.

To update your DNS record and upgrade your server installation follow these steps:

  1. Retrieve the external IP or DNS name for the traefik load balancer as described or by looking the DNS A record for app.<your domain name>` - this should already point to your traefik load balancer.

  2. Locate the DNS A record that points to the domain name of your server installation (not the one pointing to the app. subdomain)

  3. Edit the A record so that it points to the traefik load balancer, just like the record for the `app. subdomain does. Your changes might need a couple of minutes to take effect, depending on you DNS service.

Next, from the KOTS admin console dashboard, select Version History from the menu bar and click Deploy for server v3.1.0.

New features

  • Telegraf plugins can now be added to server and customized to use third party monitoring solutions, for example, Datadog. For more information, see the Metrics and Monitoring doc.

  • The option to use only private load balancers has been introduced for customers who want a fully private installation. For more information see the Load Balancers guide.

  • Server 3.x hosts build artifacts, test results, and other state in object storage. We support any S3-compatible storage and Google Cloud Storage. For more information, see the Installation guide for further information.

  • Dynamic config via setup workflows is now available on server installations. For more information see our blog post and the Dynamic Configuration docs page.

  • Runner is now available on server. For further information, including installation steps, see the Runner docs. Runner allows the use of the macOS executor in server installations and VM service functionality for customers with server installed in a private data centre.

  • The frontend load balancer from v3.0 has been removed and replaced with an Ingress resource and the Traefik Ingress controller. This is a breaking change requiring you to reconfigure your DNS. See the What’s New in server docs for further information and guidance.

  • The following services can now be externalized. For setup information, see the server v3.x installation guide:

    • Postgres

    • MongoDB

    • Vault

  • Backup and restore functionality is now available. For more information see the Backup and Restore guide.

  • Prometheus is now deployed by default with server to monitor your cluster health and usage. Prometheus can be managed and configured from the KOTS admin UI. For further information, see the Metrics and Monitoring doc.

  • Server now supports the 2XL resource class. The Nomad cluster needs to be made large enough to account for larger resource classes.

  • The lifecycle of build artifacts and test results can now be configured from the KOTS admin console under Storage Object Expiry, including the option to disable the expiration and retain artifacts and test results indefinitely.

Fixes

  • Resolved a collection of bugs that were causing sensitive information to be leaked into CircleCI support bundles:

  • Instances of faulty and partial redactions of secrets were detected, in part due to 3rd party bugs.

  • PostgresDB leaking sensitive information to STDOUT.

  • Several CircleCI services were logging secrets.

  • Tightened network security in the Nomad terraform module.

  • Terraform v0.15.0 and up are now supported.

  • Updated installation scripts to use functions supported by most recent versions of Terraform.

  • Resolved a bug that was leading to machine large builds being run on the wrong machine type. Machine large builds now correctly use 4 vCPUs and 16GB of RAM.

  • Resolved a bug that caused contexts-service to fail on expiration of Vault client tokens.

  • Resolved a bug that was causing legacy-notifier to report readiness prematurely.

  • The JVM heap size parameter has been removed for all services. The heap size is set to be half of the memory limit.

  • Changes to networking config and certs are now picked up automatically by Traefik. Previously, a restart would have been required.

  • Minimum requirements for CPU and memory have changed. For the new values, see the Installation Prerequisites doc.

Known issues

  • It is currently possible for multiple organizations under the same CircleCI server account to have contexts with identical names. This should be avoided as doing so could lead to errors and unexpected behavior.

  • CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

Release 3.0.2

  • Resolved a bug relating to artifacts disappearing after 30 days. The default settings for the artifact retention period have been updated to unlimited, and can be adjusted from the KOTS Admin Console.

  • Resolved a bug that made Traefik "unaware" of TLS certificate updates without requiring a manual restart of the Traefik pod. The Traefik pod will now restart automatically after any TLS certificate updates go into effect after the initial post KOTS deployment.

  • Resolved a bug in builds-service that was causing pods to crash as a result of running out of memory.

Release 3.0.1

  • build_agent version value updated, as the previous version was relying on a vulnerable version of PsExec.

  • Due to an issue that was causing duplicated checks in GitHub, environment variables for output-processor were reconfigured.

  • Adjusted deployment configuration for vm-service to handle out-of-order database migrations managed by Flyway.

To learn more about Server v3.x, see the following:



Help make this document better

This guide, as well as the rest of our docs, are open-source and available on GitHub. We welcome your contributions.