Changelog

What’s New in Release 2.19.04

Fixes

• Fixed a bug that was leading to support bundle creation timing out before the Services machine logs had been created, leaving only Replicated logs.

• S3 connection pool metrics under circle.s3.connection_pool.* have been added to the test results service, making it easier to debug issues with this service.

• Add in missing environment variables for the workflows service. The absence of these environment variables was causing excessive stack tracing whenever workflows were run. This in turn lead to excessive log rotation.

• Fixed a bug that was causing failure to update GitHub statuses. Some customers experienced this bug when a project had a follower with a broken auth token.

Known Issues

• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds:
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if aplicable) or private IP of the nomad client

What’s New in Release 2.19.03

Before Upgrading

If you are using an IAM role scoped to a non-root path, you will need to unset the OUTPUT_PROCESSOR_USE_NAIVE_ROLE_MAPPING environment variable in your output processor customization script. See the Customizations Guide in our documentation for more information on using customization scripts.

Fixes

• Removed the use of the depecated GitHub.com API endpoint GET applications/%s/tokens/%s.

• Distributed tracing is now enabled by default for Server installations. Traces are used in CircleCI support bundles to improve our ability to troubleshoot Server issues. Options for the tracing sampling rate are displayed in the Replicated Management Console, but should only be changed from the default if requested by CircleCI Support.

• Fixed an issue that was preventing restore_cache from working with the storage driver set to “none” - i.e not S3.

• Fixed an issue that was preventing the output_processor service from using AWS AssumeRole when the role was located in a subfolder. This issue affected customers with security policies forcing the use of a subfolder in this case, and the symptoms included the inability to store artifacts or use timings-based test splitting.

• JVM heap size can now be changed using the JVM_HEAP_SIZE environment variable for the following services: vm-service, domain-service, permissions-service and federations-service.

Known Issues

• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds:
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if aplicable) or private IP of the nomad client

What’s New in Release 2.19.02

Fixes

• In the LDAP login flow we now use an anonymous form to POST LDAP auth state, rather than sending it as a GET parameter. Previously, when a user authenticated using LDAP, their username and password were sent in plaintext as part of a query parameter in a GET request. As requests are over HTTPS, this left usernames and passwords in request logs, etc. This issue is now fixed.

• Optimizely and Zendesk are now removed from Server release images.

• Fixed an issue in which setting CIRCLE_ADMIN_SERVER_HTTP_THREADS or CIRCLE_PUBLIC_FACING_SERVER_HTTP_THREADS too high would prevent the frontend container from starting.

• Due to changes in the GitHub API we have removed the use of ?client_id=x&client_secret=y for GitHub, and GHE versions 2.17 and later.

• Fixed an issue that was causing intermittent failures to spin up VMs with DLC in use.

• Fixed an issue that was preventing the customization of proxy settings for Docker containers. See the Nomad Client Proxy and Service Configuration Overrides guides for more infomation.

• Fixed a bug that was preventing job steps for non-failing builds being logged when proxy settings were used for the job container.

• Removed legacy TLS versions 1.0 and 1.1, in addition, enabled 1.2 and 1.3 TLS, and specified the following ciphersuites
  • ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384
• Fixed a statsd configuration issue that meant some services were not emitting Telegraf metrics.

Known Issues

• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds:
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if aplicable) or private IP of the nomad client
• Classic Load Balancer is no longer available from this version due to the ciphersuite changes listed above. CircleCI no longer accepts requests from Classic Load Balancer, so you should move to Network Load Balancer (NLB) or Application Load Balancer (ALB).

What’s New in Release 2.19.01

Fixes

• Fixed a bug that was preventing some customers from upgrading due to a schema change in one of our library dependencies.

• Fixed a bug that was preventing some customers from inspecting builds via SSH due to a logic change in our build agent.

Known Issues

• If you are using an HTTP proxy for your installation, do not upgrade to v2.19.01, instead upgrade to at least v2.19.02. We have known issues around job step logging and docker commands customization that makes the v2.19.01 release incompatible with an HTTP proxy setup of CircleCI Server.

• If any changes have been made to your networking configuration you may need to run the following steps to ensure you can use SSH to inspect your builds:

  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if aplicable) or private IP of the nomad client

What’s New in Release 2.19.00

Requirements

• Before upgrading to Server v2.19 your Nomad launch configuration must be updated by following this guide. Server v2.19 uses Nomad version 0.9.3, so if you have externalized Nomad, contact your support engineer before upgrading.

New Features

• You can now customize resource classes for your installation to provide developers with CPU/RAM options for the Jobs they configure. For more information see our guide to customizing resource classes in Server v2.19.

• CircleCI Server installations on AWS can now be configured to work on GovCloud. Using AWS GovCloud enables you to meet a variety of regulatory requirements, such as FedRAMP, ITAR, DoD SRG, CJIS, and HIPAA. Read more about how CircleCI helps customers meet security and compliance needs here.

Upgrades and Fixes

• For v2.19 we have updated Nomad clusters from 0.5x to 0.9, reducing the accumulation of dead jobs and improving overall performance. For more information, see the Nomad changelog.

• The image used to run the RabbitMQ server has been updated to fix vulnerabilities.

• The Server-Mongo-Upgrader service has been removed from the Services machine. Server Mongo Upgrader was a migration service written to facilitate the upgrade from Mongo 3.2 to 3.6, which was done as a part of the CircleCI Server v2.15.0 release in October 2018. This means it is not possible to upgrade directly from a Server version pre v2.15.0 to v2.19.0.

Known Issues

• If you are using an HTTP proxy for your installation, do not upgrade to v2.19.x until further notice. We have known issues around job step logging and docker commands customization that make the current 2.19.x releases incompatible with an HTTP proxy setup of CircleCI Server. This issue will be fixed in a future 2.19.x patch release and the ‘safe’ version will be flagged up clearly for affected customers.

What’s New in Release 2.18.03

Features

Following our release of Windows support on the hosted offering of CircleCI, running VM-based jobs in Windows environments is now supported on CircleCI Server instances running in AWS.

The version of Windows currently supported is Windows Server 2019. Check out the Customizing VM Service Images doc for the details of how to build your own custom Windows Server 2019 image, and then follow the Server examples in the Getting Started with Windows doc for instructions on how to set up a Windows job on your CircleCI Server installation.

Deprecation

• Server-Mongo-Upgrader will be removed in Server 2.19.0. Only effects customers running < 2.15.0.

What’s New in Release 2.18

Upgrading your installation

Our upgrade guide takes you step by step through the process of upgrading to the latest version of CircleCI Server, including any checks and updates required before starting the upgrade process.

Note: If at any time your organization name has been changed, there is a script that must be run before starting the upgrade process.

Notes and Best Practices

• We now require a minimum 32GB of RAM for the Services Machine.

• We have made some changes to our Redis configuration. If you have externalized Redis then you’ll need to update your configuration. Please contact your Customer Success Manager.

• We have also made changes to our Postgres version and require at least postgreSQL v9.5.16. If you have externalized postgreSQL then please update to at least that version in 2.17.x before upgrading to 2.18.

Known Issues

Nomad garbage collection only runs every 4 hours. This could prevent nomad jobs from running in environments with heavy usage. If you hit this bug please follow this work around:

1. Create /etc/circleconfig/schedulerer/customizations on the services machine
2. Inside /etc/circleconfig/schedulerer/customizations enter the following:

   export SCHEDULERER_NOMAD_BACKPRESSURE_MAX_PENDING_JOBS=500 
   export SCHEDULERER_NOMAD_BACKPRESSURE_MAX_DEAD_JOBS=100000
   

3. Restart the picard-scheduler container:

   sudo docker restart picard-scheduler
   

Features/Improvements

• It is now possible to restrict environment variables at run time by adding security groups to contexts. Security groups are defined as GitHub teams or LDAP groups. After a security group is added to a context, only CircleCI users who are members of that security group may access or use the environment variables of the context. For more information see our guide to restricting a context.

• Customers running storage drivers external to AWS will see improved routing times when searching for build Artifacts.

• You can now customize the metrics that get output from CircleCI. For steps and options, see our Custom Metrics guide. Below is a short list of metrics that are included by default when enabling Custom Metrics:

• You can now provide individual AMIs for both Remote Docker and machine executor jobs. Previously we provided the option for a single custom AMI to be used across both, but with v2.18, this expanded customization gives you greater control over versioning and dependencies to meet your individual CICD needs. See our VM Service guide for more information.

Bug fixes

• Additional fixes around contexts and org renames.

• Fixed an issue where occasionally volumes would fail to attach to spun up Remote docker/machine instances.

• Fixed an issue where the CircleCI integration could not be installed on JIRA instances with the jira.com subdomain.

• Fixed an issue where the Workflows page would still point to an old repo after renaming an organization.

• Fixed issue where the Workflows UI would fail to refresh data automatically.

• Improved context loading times in cases when they could cause timeouts in the UI.

• Fixed an issue where contexts would cause builds to return CIRCLE_BUG.

What’s New in Release 2.17

Known Issues

There have been reports of our backpressure settings being too aggressive due to lack of nomad garbage collecting, as well as issues with the Vm Service overwhelming AWS API Limits. If you are on 2.17, you should upgrade to 2.17.1.

Required Changes

IMPORTANT We have upgraded Replicated to version 2.34.1 which requires Docker 17.12.1. Please follow the instructions provided in the Updating Server Replicated Version guide before upgrading.

• When using Github Enterprise, operators will now need to provide CircleCI with a Github User Token in the Management Console Settings. This is required due to a change in how the Github Enterprise API allows us to verify Organizations. It is recommended that you use a shared ops account to generate this token. After upgrading to CircleCI Server v2.17, please follow these steps to enact this change:
  • Sign into your GitHub Enterprise instance.
  • Navigate to Personal Settings (top right) > Developer Settings > Personal Access Tokens.
  • Click “generate new token”. Name the token appropriately to prevent accidental deletion. Do not tick any of the checkboxes.
  • Copy the new token and paste it into the Management Console settings. (Github Integration > GitHub Enterprise Default API Token)
  • Save the settings in Replicated.

Notes and Best Practices

• Fill out this form to receive updates about CircleCI through email.
• It is currently a best practice to use a Services Machine with a minimum of 32GB of RAM. Starting in 2.18 it will become required. See docs for our recommendation(s).
• We have updated our software packages to the following versions. This is not a required update for those with externalized environments at this time, but will be when v2.18 is released.
  • Vault
    • 1.1.2
  • Mongo
    • 3.6.12-xenial
  • Redis
    • 4.0.14
• We are removing the 1.0 Single-Box options from CircleCI 2.0. We found a few critical vulnerabilities in our 1.0 build image, and we have long stopped recommending it for trials. If this is absolutely critical to your workflow please reach out to us. This does not impact people who are running 1.0 in clustered mode.

Features/Improvements

• Workflows now has a Slack Integration! Users can choose to receive Slack notifications when their workflows complete.
• Administrators can now restrict what organizations are allowed into their CircleCI installation. For more details on how to enable this feature, please see the User Management Section of the 2.17 Ops Manual.
• We changed the renamed org flow so orgs that have been renamed will no longer result in errors going forward. Existing users who had applied workarounds for this use case will now no longer require said workarounds.
• Workflows now take up less DB space and will be easier to maintain going forward.
• Improved the cache in front of GraphQL API resulting in overall improved performance.
• Added backpressure to avoid overwhelming nomad with requests, this will result in increased performance from existing nomad clusters.

• New machine executor AMIs based on Ubuntu 16.04 for AWS.

• Ubuntu 16.04 with Docker 18.06.3 has apt-daily and apt-daily-upgrade services disabled.
  • It is highly recommended that customers try to experiment with the AMIs below before officially switching:
  • The new images are as follows
    • Ap-northeast-1:ami-0e49af0659db9fc5d
    • Ap-northeast-2:ami-03e485694bc2da249
    • Ap-south-1:ami-050370e57dfc6574a
    • Ap-southeast-1:ami-0a75ff7b28897268c
    • Ap-southeast-2:ami-072b1b45245549586
    • Ca-central-1:ami-0e44086f0f518ad2d
    • Eu-central-1:ami-09cbcfe446101b4ea
    • Eu-west-1:ami-0d1cbc2cc3075510a
    • Eu-west-2:ami-0bd22dcdc30fa260b
    • Sa-east-1:ami-038596d5a4fc9893b
    • Us-east-1:ami-0843ca047684abe87
    • Us-east-2:ami-03d60a35576647f63
    • Us-west-1:ami-06f6efb13d9ccf93d
    • us-west-2:ami-0b5b8ad02f405a909
  • They are replacing:
    • Ap-northeast-1:ami-cbe000ad
    • Ap-northeast-2:ami-629b420c
    • Ap-south-1:ami-97bac2f8
    • Ap-southeast-1:ami-63b22100
    • Ap-southeast-2:ami-dd6c73be
    • Ca-central-1:ami-d02c93b4
    • Eu-central-1:ami-b243eedd
    • Eu-west-1:ami-61de3718
    • Eu-west-2:ami-904e5ff4
    • Sa-east-1:ami-c22057ae
    • Us-east-1:ami-05b6e77e
    • Us-east-2:ami-9b4161fe
    • Us-west-1:ami-efc9e08f
    • Us-west-2:ami-ce8c94b7

Bug fixes

• Fixed some bugs related to GitHub API response handling and webhook handling.
• Fixed issue with Scheduled Workflows when the services machine is restarted.
• Fixed changing the RabbitMQ hostname for Scheduled Workflows when externalizing.
• You can no longer create contexts with no names. If you are using a context with no names, it will no longer be accessible from your execution environment.
• We have optimized our handling of large amounts of build output and test results XML, to avoid out-of-memory errors.
• The CIRCLE_PULL_REQUEST environment variable was not being populated correctly in all cases when building across forks. This has been fixed.
• Fixed a bug where some commits with [ci skip] in the message were still being built.
• Fixed a bug causing Workflows to get stuck when infrastructure_failure happens after a job fails.
• Fixed a bug causing duplicate docker networks on same nomad client (if running build using machine:true AND vm-provider=on_host).
• Improved performance when using local storage. Previously, caching issues had been experienced when local storage was used rather than the default option of using S3 (selecting None under Storage Driver options from the Management Console).
• We have added more error checking and validation around Github’s API so the existing list commit endpoint no longer causes issues.
• Datadog API token field was stored in plaintext, now set as a password field.
• Fixed issue where workflows were constrained from fanning out to large number of jobs.

Fill out this form to receive updates about CircleCI through email.

What’s Fixed in Release 2.16.1

• Bug Fix: This release includes a new flag to disable GitHub cache warming. Note: Only add this flag if a high thread count is impacting the API service. To disable cache warming after completing the upgrade, complete the following steps:

1. Add the export DOMAIN_SERVICE_REFRESH_USERS=false flag to the /etc/circleconfig/api-service/customizations file on the Services machine.

2. Restart CircleCI by going to the Administrative console and clicking Stop Now, waiting for it to stop, then clicking Start.

• Bug Fix: Fixed issue preventing metrics from domain service from being forwarded.

• Security Fix: The SMTP password field in the dashboard was set to plain text. The password field will now be hidden.

Fill out this form to receive updates about CircleCI through email.

Note: It is currently a best practice to use a Services Machine with a minimum of 32GB of RAM. Starting in 2019 it will become required. See docs for our recommendation(s).

What’s New in Release 2.16

The following updates and fixes are also of note:

IMPORTANT We have upgraded Replicated to version 2.29.0 which requires Docker 17.12.1. Please follow the instructions provided in the Updating Server Replicated Version guide before upgrading.

New Feature We are excited to announce that you can now distribute your data and workload external to the Services Machine. The following services can be externalized; MongoDB, Redis, Nomad Server, RabbitMQ, Postgres and Vault. Please contact your CSM for the latest documentation.

New Feature Custom Metrics can now be accomplished via a Telegraf Output Configuration File. See the Custom Metrics section of the Monitoring section. Note: The Custom Metrics section in the admin UI been disabled in favor of a configuration file.

New Feature Users can now receive email notifications about Workflows

New Feature The PostgreSQL image has been updated to allow modifying the default configuration by creating the following file: /etc/circleconfig/postgres/extra.conf A list of configuration options can be found here.

Updated Removing EOL Banner on build emails

Updated VM Service stability improvements

Bug Fix Fix issue with contexts that would break after 32 days

Bug Fix Vault auth tokens will now renew periodically to prevent expiration

Bug Fix Fix for an issue where workflow listing pages would intermittently fail to render

Bug Fix We have fixed a small number of bugs affecting processing incoming web hooks from GitHub to CircleCI.

Bug Fix For installs that are using an HTTP or HTTPS proxy, the jobs will now ignore that proxy so that setup_remote_docker works.

Bug Fix Reduced the number of connections 1.0 builders make to the PostgreSQL database.

Security Fix Security fixes for potential cross-site scripting vulnerability and HTTP header injection vulnerability

Security Fix Forked PRs can no longer write the caches of parent projects by default for security reasons. It is possible to still write parent project caches from the fork if the “Pass secrets to builds from forked pull requests” (in Advanced settings) is enabled.

Update Improved metrics for VM provisioning for machine executor. This changed metric names, so if you are monitoring VM provisioning in your install already, you will need to reconfigure the monitoring dashboards for these new metrics.

• New metrics:
  • vm-service.gauges.available-vms and vm-service.gauges.running-vms replaced by vm-service.gauges.vms_by_status
  • vm-service.gauges.running-tasks replaced by vm-service.gauges.tasks_by_status
  • vm-service.gauges.oldest-unassigned-task replaced by vm-service.gauges.unassigned_tasks_age

Known Issue(s)

When a saved context is removed during the build graph phase it can cause a silent failure during job execution.
In some instances the metrics section becomes hidden in the settings page. Please see this Knowledge article for instructions.

Fill out this form to receive updates about CircleCI through email.

Note: It is currently a best practice to use a Services Machine with a minimum of 32GB of RAM. Starting in 2019 it will become required. See docs for our recommendation(s).

What’s New in Release 2.15

This release contains an upgrade to MongoDB, service migrations for contexts and is a required upgrade. The following updates and fixes are also of note:

MongoDB Upgrade The MongoDB version running on Services VM will be automatically updated from 3.2 to 3.6 for you. If you have externalized MongoDB and wish to upgrade to 3.6, please contact your CSM

Bug Fix Restrict interaction of Workflows ‘Rerun’ dropdown to users who are part of GitHub organization for open-source projects.

Bug Fix Known issue for broken UI due to workflows with “.” in the name

Bug Fix Known performance issue with freezed browsers for organizations with many branches.

Bug Fix Known issue for Vault auth tokens not being automatically renewed. If you have externalized Vault, please contact your CSM before you upgrade to 2.15.0

New Feature: Splitting test files via Workflows. If your project has a large number of tests, you can use circleci tests split feature to reduce your job run time. For more information on how to setup tests split, please follow this document.

New Feature: Workflows UI will now show actors who have taken the following actions:

• Rerun workflows
• Approved jobs
• Canceled jobs

New Feature We are excited to announce more Metrics! Nomad job metrics can be enabled and will be emitted by the Nomad Server. Please see Nomad Metrics documentation for instructions on how to configure Nomad Clients. The following five types of metrics are reported:

• circle.nomad.server_agent.poll_failure: Returns 1 if the last poll of the Nomad agent failed, otherwise it returns 0.
• circle.nomad.server_agent.jobs.pending: Returns the total number of pending jobs across the cluster.
• circle.nomad.server_agent.jobs.running: Returns the total number of running jobs across the cluster.
• circle.nomad.server_agent.jobs.complete: Returns the total number of complete jobs across the cluster.
• circle.nomad.server_agent.jobs.dead: Returns the total number of dead jobs across the cluster.

New Feature Host (Services VM) and docker metrics can now be forwarded via Telegraf , a plugin-driven server agent for collecting & reporting metrics, to DataDog or a custom location. To enable and/or configure please go tohttps://example.com/settings#metrics. For more information please see the Monitoring documentation.

Performance Improvement Reduced the load time for all workflows pages.

Performance Improvement CircleCI has upgraded the mechanism that returns gettimeofday calls. As a result, heavy users of syscall will see an aggregate improvement in performance. Most noticeably this will improve the run time of projects with heavy usage of debugging/profiling tools and Xdebug with PHP.

Updated JVM_HEAP_SIZE is configurable for frontend, output-processor, and test-result containers. Please see JVM Heap Size Configuration documentation.

Updated For customers running the Static installation, Ubuntu 16.04 (Xenial) has been added as a supported OS. For additional details please see the docs.

Known Issue(s)

When a saved context is removed during the build graph phase it can cause a silent failure during job execution.
When using the none storage driver, the cache will not be saved. This means that you currently cannot use caching in CircleCI when you are using the none storage driver.

What’s New in Release 2.14

This release contains service migrations for contexts and is a required upgrade. The following updates and fixes are also of note:

Fixed: Pre-allocated VMs not being used. Customers reported that jobs using machine executor and setup_remote_docker were continuing to spin up (and terminate) new VMs at runtime instead of the pre-allocated pool.

Fixed: Unable to build in AWS region eu-west-2.

Updated: The title of the Builds page in the CircleCI application was changed to Jobs in several places to better reflect the structure of builds.

Updated: Contexts are now checked for unique naming across your installation of CircleCI instead of just within an organization. You will no longer be able to create duplicate context names within an account. Existing duplicates will be honored by the system, but should be changed to avoid future collisions. Please see the Contexts documentation for further information.

Updated: The test-results services cleans up the /tmp directory on restart and the JVM heap size increased from 256m to 512m.

Known Issue(s)

When a saved context is removed during the build graph phase it can cause a silent failure during job execution.
When using the none storage driver, the cache will not be saved. This means that you currently cannot use caching in CircleCI when you are using the none storage driver.

Migration Tools: The 1.0 migration center was updated with additional content and helpful tools to migrate projects from CircleCI 1.0 to 2.0.

What’s New in Release 2.13*

• Fixed: Due to a certificate not being imported jobs were unable to submit after a container restart.
• Fixed: Stability issues within contexts service.
• Fixed: vm-service issue where AWS ECS and EBS volumes were left used. Added better idempotent client retries, API eventual consistency handling.
• Added: vm-service EBS Volumes cache evictor. Docker Layer Caching EBS volumes get deleted after 14 days of unused. You can control the inactivity period in Installation Admin Settings page.
• Added: Periodic checks to scale down any unused long running instances (e.g. due to reduction in prescaling configurations, or or due to a bug preventing termination of VMs in timely manner).
• Added: Optimizations to speed up provisioning. Resiliency to network/external changes.
• CircleCI is excited to announce that users on AWS can now forward host (Services VM) and docker metrics via Telegraf , a plugin-driven server agent for collecting & reporting metrics, to AWS CloudWatch. This is is the first step into bringing you greater observability into your installation. To enable and configure please go tohttps://example.com/settings#metrics. In this release the following metrics may be enabled: Docker , Networking , Disk , Memory, and CPU. For more information please see the Monitoring documentation.
• CircleCI is enhancing visibility by capturing usage statistics of CircleCI installations. This aggregate data will be used to better support you and ensure a smooth transition off 1.0. This optional feature is automatically available when you upgrade to the latest version of your installation or once a support bundle is downloaded. Below are the metrics being captured. If you would like access to this data in order to better understand your users please contact a CircleCI account representative. Please reference exhibit C within your terms of contract and our standard license agreement for our complete security and privacy disclosures. For more information please see the Usage Statistics documentation.

Weekly Account Usage

Weekly Job Activity

* this release contains migrations and is a required upgrade

The installable version of CircleCI Server 2.9.0 includes the following improvements and bug fixes:

• Resolution for a potential LDAP injection vulnerability
• Addition of support for the following LDAP fields:
  • Group Membership
  • Group Object Classes

See the Authentication document for the steps.

• Addition of default values for OpenLDAP and Active Directory
• Stability and bug fixes for an issue with test results
• Improvements to ready agent logging for support bundles
• Stability fixes and speed improvements for provisioning remote Docker engine
• Reliability improvements for the permissions service client
• Differentiation between the LDAP login and email to remove email as the assumed login value
• Fix for a bug that deselected all projects when a new user logged in for the first time
• Fix for a bug that was preventing Workflows from working correctly when located behind a proxy
• Bug fixes and improvements for Contexts

The installable version of CircleCI 2.8.0 includes the following fixes and improvements:

• An issue involving pre-allocated remote docker, machine VM creation, and other VM usage on AWS when the on-host VM option is selected has been fixed.

• An issue with the VM provider timing out when waiting for SSH to connect has been fixed. This upgrade supersedes any custom values for the Number of SSH attempts for starting VM setting and adds an equivalent field named Timeout for waiting for SSH to connect to VM(s) that you must set.

The installable version of CircleCI 2.6.0 includes the following fixes and improvements:

• It is now possible to create multiple Contexts in the Organization Settings of the CircleCI app. See the Contexts documentation for details and naming conventions.

The installable version of CircleCI 2.5.0 includes the following fixes and improvements:

• This release fixes a problem with DNS resolution failure on alpine-based containers. This problem was observed in the 2.3 release and was fixed for some service containers in 2.4. The 2.5 update fixes the issue for all service containers.
• This release includes improvements to scaling and garbage collection in the VM Service that should improve performance for installations with heavy use of machine executors or remote Docker.

The installable version of CircleCI 2.4.0 includes the following improvements and changes:

• Audit logging - This feature makes available a downloadable log of system events from the CircleCI Admin dashboard. This is the first release of audit logging; more events and improved data in the logs will be delivered in subsequent releases. Find detailed information about the data fields for each event in the Audit Logging section of the Security Features document.

• LDAP integration - It is possible to enable a new CircleCI installation to authenticate users with OpenLDAP or Active Directory credentials. New installations may be configured to have users authenticate through LDAP, including Active Directory support. Refer to the Authentication document for instructions.

• New build-agent - The new build-agent fixes some cases in which users were unable to log into a running job with SSH.

• VM Service improvements - Autoscaling of the VM Service has been improved, so machine executors should have more consistent performance at scale.

The installable version of CircleCI 2.2.0 includes the following improvements and changes:

• New visualization for Workflows - When you click on a workflow in the CircleCI application, a diagram appears with the relationship of your jobs to one another and the complete sequence of the workflow.
• Updates to the onboarding flow - We now show a new user only projects already building on CircleCI rather than a list of all projects found in source control. This change was made to prevent inadvertent adding of projects to CircleCI during the onboarding process. We also added a confirmation dialog for any user opting to follow more than six projects and made it clearer how to add new projects not yet building on CircleCI.
• Support for object stores other than S3 - CircleCI 2.2.0 now supports the same object store options that are supported in 1.x. This is a prerequisite to future releases that will allow non-AWS installations of CircleCI 2.x.
• Updates to the VM Service; ACTION MAY BE REQUIRED IF YOU ARE USING AWS - The VM service has been updated to use volumes rather than persistent instances. This change allows operators to accommodate Docker Layer Caching with a smaller number of long-running VMs. If you installed using the suggested IAM permissions from prior versions you must update your IAM permissions with this upgrade. Refer to the policy template for the complete set of permissions.
• New Contexts feature; ACTION MAY BE REQUIRED IF YOU ARE USING THE HIGH AVAILABILITY CONFIGURATION - The Contexts feature uses a HashiCorp Vault secure storage mechanism. If your installation is running in HA mode, HashiCorp Vault needs to be externalized similar to the Mongo and Postgres databases. Refer to the Configuring High Availability documentation for instructions.

CircleCI 2.0 is a completely updated CI/CD platform that starts every run with a clean image which is automatically provisioned just-in-time for Linux jobs on the CircleCI application installed in your private cloud. The CircleCI 2.0 platform includes significant performance, stability, and reliability improvements along with first-class Docker support, Workflows (pipelines), Resource Allocation, and Contexts. The Management Console (Replicated) also includes new options for enabling 2.0 Builders and appropriate default settings.

If you have CircleCI installed you may access CircleCI 2.0 features on your current installation with no restrictions under your current agreement and support level. However, you must contact your CircleCI account representative for assistance with upgrading, to obtain migration scripts, and to receive a new license channel.

• CircleCI 2.0.0 is only available on AWS and may be installed with Terraform or manually.
• Teams may create a CircleCI 2.0 .circleci/config.yml file in their repositories to add new 2.0 projects incrementally while continuing to build 1.0 projects which use a circle.yml configuration file.

Documentation for the installable 2.0 application is available at CircleCI 2.0 Documentation for the following topics:

• Administrator’s Overview
• Administration FAQ
• Security Features
• Installing CircleCI 2.0 on Amazon Web Services with Terraform
• Configuring High Availability
• Administrative Variables, Monitoring, and Logging
• Introduction to Nomad Cluster Operation
• Backing up CircleCI Data

This is a security update to prevent logging temporary access tokens in build output. Those access tokens would have allowed a user to do a PUT request to overwrite an existing artifact only if all of the following were true:

1. They already had access to the build.
2. They identified the token in the build output.
3. The were to use the token within 30 minutes after the original artifact was uploaded.

The installable version of CircleCI 2.3.0 includes the following improvements and changes:

• Users can now schedule workflows. The underlying cron-service has been added to the installation, though no specific configuration should be needed. See the documentation on Workflows Orchestration for details.
• Support for using private IP addresses in 1.0 builds is now enabled along with 2.0 builds
• A bug in the VM garbage collector was addressed that should help in situations when VMs were being leaked.
• We have disabled Replicated snapshots. Please see the documentation on backups if you have previously used Replicated snapshots.

We have released patches to our AMIs and other infrastructure to address CVE-2016-8655. We recommend all CircleCI Enterprise installations follow the instructions below to update both their Services box and their Builder fleet.

If you have any questions or difficulties please contact enterprise-support@circleci.com.

Update the Services box:

1. As always, ensure your data is backed up.
2. Shut down CircleCI in the Replicated console (or via the CLI).
3. Update the kernel using the provided install_kernel_master function below.
4. Restart the machine.

#!/bin/bash
function install_kernel_master() {
    echo '>>> Installing Kernel'
    apt-get update
    apt-get install linux-image-3.13.0-105-generic linux-headers-3.13.0-105-generic linux-image-extra-3.13.0-105-generic
    apt-cache policy linux-image-3.13.0-105-generic linux-headers-3.13.0-105-generic linux-image-extra-3.13.0-105-generic
}

Update the Builder fleet:

For the builder fleet, update the Launch Configuration to use the updated AMI from the list below:

• ap-northeast-1 = “ami-07f09d60”
• ap-northeast-2 = “ami-90588ffe”
• ap-southeast-1 = “ami-4f54f82c”
• ap-southeast-2 = “ami-040c3467”
• eu-central-1 = “ami-3465a35b”
• eu-west-1 = “ami-d70421a4”
• sa-east-1 = “ami-2e6bf242”
• us-east-1 = “ami-e68f89f1”
• us-west-1 = “ami-901c4af0”
• us-west-2 = “ami-0c57fc6c”

If you are using our Terraform scripts, you can download the new script https://github.com/circleci/enterprise-setup/blob/master/circleci.tf and run terraform apply. We’ve already updated the scripts to include the new AMIs, so terraform should launch new builders automatically with the patched version, and cycle your fleet.

If you are using a non-AWS environment, use the same method to patch your builders you used to patch the Services box.

If any of the above does not apply to your environment, or you encounter issues with your upgrades please contact: enterprise-support@circleci.com.