サーバーの更新履歴

What’s New in Release 2.19.07

Fixes

• Fixed a bug that was preventing authentication to China region ECR images in the docker executor.
• Removed the recursive chown from startup of the fileserverd service. In instances of heavy usage this was causing the startup process to take a long time, or in some cases, startup was blocked.

Known Issues

• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds (for further information on SSH reruns see the SSH Rerun Guide):
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if applicable) or private IP of the nomad client

What’s New in Release 2.19.06

Fixes

• Fixed a bug that was causing workflow statuses to be displayed on GitHub in the wrong order.
• Introduced performance improvements to workflows-conductor that dramatically reduce CPU usage and latency.
• Fixed a bug that was preventing use of an S3 storage region other than us-east-1 via an IAM user.
• Fixed formatting type of SMTP passwords to ensure they are masked during setup.

Known Issues

• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds (for further information on SSH reruns see the SSH Rerun Guide):
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if aplicable) or private IP of the nomad client

What’s New in Release 2.19.05

Fixes

• Fixed a bug that could lead to the VM database ending up in an incorrect state in the event of a Services Machine crash or manual termination of VM Service instances.

Known Issues

• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds:
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if aplicable) or private IP of the nomad client

What’s New in Release 2.19.04

Fixes

• Fixed a bug that was leading to support bundle creation timing out before the Services machine logs had been created, leaving only Replicated logs.

• S3 connection pool metrics under circle.s3.connection_pool.* have been added to the test results service, making it easier to debug issues with this service.

• Add in missing environment variables for the workflows service. The absence of these environment variables was causing excessive stack tracing whenever workflows were run. This in turn lead to excessive log rotation.

• Fixed a bug that was causing failure to update GitHub statuses. Some customers experienced this bug when a project had a follower with a broken auth token.

Known Issues

• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds:
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if aplicable) or private IP of the nomad client

What’s New in Release 2.19.03

Before Upgrading

If you are using an IAM role scoped to a non-root path, you will need to unset the OUTPUT_PROCESSOR_USE_NAIVE_ROLE_MAPPING environment variable in your output processor customization script. See the Customizations Guide in our documentation for more information on using customization scripts.

Fixes

• Removed the use of the depecated GitHub.com API endpoint GET applications/%s/tokens/%s.

• Distributed tracing is now enabled by default for Server installations. Traces are used in CircleCI support bundles to improve our ability to troubleshoot Server issues. Options for the tracing sampling rate are displayed in the Replicated Management Console, but should only be changed from the default if requested by CircleCI Support.

• Fixed an issue that was preventing restore_cache from working with the storage driver set to “none” - i.e not S3.

• Fixed an issue that was preventing the output_processor service from using AWS AssumeRole when the role was located in a subfolder. This issue affected customers with security policies forcing the use of a subfolder in this case, and the symptoms included the inability to store artifacts or use timings-based test splitting.

• JVM heap size can now be changed using the JVM_HEAP_SIZE environment variable for the following services: vm-service, domain-service, permissions-service and federations-service.

Known Issues

• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds:
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if aplicable) or private IP of the nomad client

What’s New in Release 2.19.02

Fixes

• In the LDAP login flow we now use an anonymous form to POST LDAP auth state, rather than sending it as a GET parameter. Previously, when a user authenticated using LDAP, their username and password were sent in plaintext as part of a query parameter in a GET request. As requests are over HTTPS, this left usernames and passwords in request logs, etc. This issue is now fixed.

• Optimizely and Zendesk are now removed from Server release images.

• Fixed an issue in which setting CIRCLE_ADMIN_SERVER_HTTP_THREADS or CIRCLE_PUBLIC_FACING_SERVER_HTTP_THREADS too high would prevent the frontend container from starting.

• Due to changes in the GitHub API we have removed the use of ?client_id=x&client_secret=y for GitHub, and GHE versions 2.17 and later.

• Fixed an issue that was causing intermittent failures to spin up VMs with DLC in use.

• Fixed an issue that was preventing the customization of proxy settings for Docker containers. See the Nomad Client Proxy and Service Configuration Overrides guides for more infomation.

• Fixed a bug that was preventing job steps for non-failing builds being logged when proxy settings were used for the job container.

• Removed legacy TLS versions 1.0 and 1.1, in addition, enabled 1.2 and 1.3 TLS, and specified the following ciphersuites
  • ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384
• Fixed a statsd configuration issue that meant some services were not emitting Telegraf metrics.

Known Issues

• If any changes have been made to your networking configuration from the default, you should run the following steps to ensure you can use SSH to inspect your builds:
  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if aplicable) or private IP of the nomad client
• Classic Load Balancer is no longer available from this version due to the ciphersuite changes listed above. CircleCI no longer accepts requests from Classic Load Balancer, so you should move to Network Load Balancer (NLB) or Application Load Balancer (ALB).

What’s New in Release 2.19.01

Fixes

• Fixed a bug that was preventing some customers from upgrading due to a schema change in one of our library dependencies.

• Fixed a bug that was preventing some customers from inspecting builds via SSH due to a logic change in our build agent.

Known Issues

• If you are using an HTTP proxy for your installation, do not upgrade to v2.19.01, instead upgrade to at least v2.19.02. We have known issues around job step logging and docker commands customization that makes the v2.19.01 release incompatible with an HTTP proxy setup of CircleCI Server.

• If any changes have been made to your networking configuration you may need to run the following steps to ensure you can use SSH to inspect your builds:

  • For customers using AWS, make sure that you have the latest Launch Configuration configured for Nomad clients, and that exiting Nomad clients were spun up using the Launch Configuration.
  • On each Nomad Client machine, create /etc/circleci/public-ipv4
  • This file should contain the public (if aplicable) or private IP of the nomad client

What’s New in Release 2.19.00

Requirements

• Before upgrading to Server v2.19 your Nomad launch configuration must be updated by following this guide. Server v2.19 uses Nomad version 0.9.3, so if you have externalized Nomad, contact your support engineer before upgrading.

New Features

• You can now customize resource classes for your installation to provide developers with CPU/RAM options for the Jobs they configure. For more information see our guide to customizing resource classes in Server v2.19.

• CircleCI Server installations on AWS can now be configured to work on GovCloud. Using AWS GovCloud enables you to meet a variety of regulatory requirements, such as FedRAMP, ITAR, DoD SRG, CJIS, and HIPAA. Read more about how CircleCI helps customers meet security and compliance needs here.

Upgrades and Fixes

• For v2.19 we have updated Nomad clusters from 0.5x to 0.9, reducing the accumulation of dead jobs and improving overall performance. For more information, see the Nomad changelog.

• The image used to run the RabbitMQ server has been updated to fix vulnerabilities.

• The Server-Mongo-Upgrader service has been removed from the Services machine. Server Mongo Upgrader was a migration service written to facilitate the upgrade from Mongo 3.2 to 3.6, which was done as a part of the CircleCI Server v2.15.0 release in October 2018. This means it is not possible to upgrade directly from a Server version pre v2.15.0 to v2.19.0.

Known Issues

• If you are using an HTTP proxy for your installation, do not upgrade to v2.19.x until further notice. We have known issues around job step logging and docker commands customization that make the current 2.19.x releases incompatible with an HTTP proxy setup of CircleCI Server. This issue will be fixed in a future 2.19.x patch release and the ‘safe’ version will be flagged up clearly for affected customers.

What’s New in Release 2.18.03

Features

Following our release of Windows support on the hosted offering of CircleCI, running VM-based jobs in Windows environments is now supported on CircleCI Server instances running in AWS.

The version of Windows currently supported is Windows Server 2019. Check out the Customizing VM Service Images doc for the details of how to build your own custom Windows Server 2019 image, and then follow the Server examples in the Getting Started with Windows doc for instructions on how to set up a Windows job on your CircleCI Server installation.

Deprecation

• Server-Mongo-Upgrader will be removed in Server 2.19.0. Only effects customers running < 2.15.0.

What’s New in Release 2.18.02

Bug Fix

• Fixed an error when setting CIRCLE_PUBLIC_FACING_SERVER_HTTP_THREADS or CIRCLE_ADMIN_SERVER_HTTP_THREADS above 50.

What’s New in Release 2.18.01

• Introduced configuration for timeouts for the following HTTP calls to GitHub (GHE):
  • GITHUB_CONN_TIMEOUT
    • Individual configuration for connection timeout, default 5000ms
  • GITHUB_SOCKET_TIMEOUT
    • Individual configuration for socket timeout, default 5000ms
  • GITHUB_CONN_REQUEST_TIMEOUT
    • Individual configuration for receiving connection from connection pool, default 5000ms
  • GITHUB_TIMEOUT
    • Single configuration for all of the above timeouts
• Improved performance and monitoring of output-proceessor

Bug fixes

• Fixed an issue with Nomad garbage collection that had the potential to prevent jobs from being run for heavy workloads. This issue was fixed in v2.17.1 but regressed in v2.18.

What’s New in Release 2.18

Upgrading your installation

Our upgrade guide takes you step by step through the process of upgrading to the latest version of CircleCI Server, including any checks and updates required before starting the upgrade process.

Note: If at any time your organization name has been changed, there is a script that must be run before starting the upgrade process.

Notes and Best Practices

• We now require a minimum 32GB of RAM for the Services Machine.

• We have made some changes to our Redis configuration. If you have externalized Redis then you’ll need to update your configuration. Please contact your Customer Success Manager.

• We have also made changes to our Postgres version and require at least postgreSQL v9.5.16. If you have externalized postgreSQL then please update to at least that version in 2.17.x before upgrading to 2.18.

Known Issues

Nomad garbage collection only runs every 4 hours. This could prevent nomad jobs from running in environments with heavy usage. If you hit this bug please follow this work around:

1. Create /etc/circleconfig/schedulerer/customizations on the services machine
2. Inside /etc/circleconfig/schedulerer/customizations enter the following:

   export SCHEDULERER_NOMAD_BACKPRESSURE_MAX_PENDING_JOBS=500 
   export SCHEDULERER_NOMAD_BACKPRESSURE_MAX_DEAD_JOBS=100000
   

3. Restart the picard-scheduler container:

   sudo docker restart picard-scheduler
   

Features/Improvements

• It is now possible to restrict environment variables at run time by adding security groups to contexts. Security groups are defined as GitHub teams or LDAP groups. After a security group is added to a context, only CircleCI users who are members of that security group may access or use the environment variables of the context. For more information see our guide to restricting a context.

• Customers running storage drivers external to AWS will see improved routing times when searching for build Artifacts.

• You can now customize the metrics that get output from CircleCI. For steps and options, see our Custom Metrics guide. Below is a short list of metrics that are included by default when enabling Custom Metrics:

• You can now provide individual AMIs for both Remote Docker and machine executor jobs. Previously we provided the option for a single custom AMI to be used across both, but with v2.18, this expanded customization gives you greater control over versioning and dependencies to meet your individual CICD needs. See our VM Service guide for more information.

Bug fixes

• Additional fixes around contexts and org renames.

• Fixed an issue where occasionally volumes would fail to attach to spun up Remote docker/machine instances.

• Fixed an issue where the CircleCI integration could not be installed on JIRA instances with the jira.com subdomain.

• Fixed an issue where the Workflows page would still point to an old repo after renaming an organization.

• Fixed issue where the Workflows UI would fail to refresh data automatically.

• Improved context loading times in cases when they could cause timeouts in the UI.

• Fixed an issue where contexts would cause builds to return CIRCLE_BUG.

What’s New in Release 2.17.3

New Features:

• Added ability to place banners on login screens. See our guide here.

What’s New in Release 2.17.2

New Features:

• Added and exposed the following metrics for operators:
  • picard.*
    • This will affect the execution services, the important ones are located in output-processor.
  • jvm.*
    • This will allow our operators to understand and monitor when certain services are running out of memory.

What’s New in Release 2.17.1

Bug fixes

• Addressed a configuration issue with VM Service running into rate limiting errors when calling out to the AWS API. This prevented VM Service from spinning up machine and remote_executors.
• Fixed issues around nomad garbage collection only garbage collecting every 4 hours. This could prevent nomad jobs from running in environments with heavy usage.

What’s New in Release 2.17

Known Issues

There have been reports of our backpressure settings being too aggressive due to lack of nomad garbage collecting, as well as issues with the Vm Service overwhelming AWS API Limits. If you are on 2.17, you should upgrade to 2.17.1.

Required Changes

IMPORTANT We have upgraded Replicated to version 2.34.1 which requires Docker 17.12.1. Please follow the instructions provided in the Updating Server Replicated Version guide before upgrading.

• When using Github Enterprise, operators will now need to provide CircleCI with a Github User Token in the Management Console Settings. This is required due to a change in how the Github Enterprise API allows us to verify Organizations. It is recommended that you use a shared ops account to generate this token. After upgrading to CircleCI Server v2.17, please follow these steps to enact this change:
  • Sign into your GitHub Enterprise instance.
  • Navigate to Personal Settings (top right) > Developer Settings > Personal Access Tokens.
  • Click “generate new token”. Name the token appropriately to prevent accidental deletion. Do not tick any of the checkboxes.
  • Copy the new token and paste it into the Management Console settings. (Github Integration > GitHub Enterprise Default API Token)
  • Save the settings in Replicated.

Notes and Best Practices

• Fill out this form to receive updates about CircleCI through email.
• It is currently a best practice to use a Services Machine with a minimum of 32GB of RAM. Starting in 2.18 it will become required. See docs%20for%20our%20recommendation(s).
• We have updated our software packages to the following versions. This is not a required update for those with externalized environments at this time, but will be when v2.18 is released.
  • Vault
    • 1.1.2
  • Mongo
    • 3.6.12-xenial
  • Redis
    • 4.0.14
• We are removing the 1.0 Single-Box options from CircleCI 2.0. We found a few critical vulnerabilities in our 1.0 build image, and we have long stopped recommending it for trials. If this is absolutely critical to your workflow please reach out to us. This does not impact people who are running 1.0 in clustered mode.

Before:

After:

Features/Improvements

• Workflows now has a Slack Integration! Users can choose to receive Slack notifications when their workflows complete.
• Administrators can now restrict what organizations are allowed into their CircleCI installation. For more details on how to enable this feature, please see the User Management Section of the 2.17 Ops Manual.
• We changed the renamed org flow so orgs that have been renamed will no longer result in errors going forward. Existing users who had applied workarounds for this use case will now no longer require said workarounds.
• Workflows now take up less DB space and will be easier to maintain going forward.
• Improved the cache in front of GraphQL API resulting in overall improved performance.
• Added backpressure to avoid overwhelming nomad with requests, this will result in increased performance from existing nomad clusters.

• New machine executor AMIs based on Ubuntu 16.04 for AWS.

• Ubuntu 16.04 with Docker 18.06.3 has apt-daily and apt-daily-upgrade services disabled.
  • It is highly recommended that customers try to experiment with the AMIs below before officially switching:
  • The new images are as follows
    • Ap-northeast-1:ami-0e49af0659db9fc5d
    • Ap-northeast-2:ami-03e485694bc2da249
    • Ap-south-1:ami-050370e57dfc6574a
    • Ap-southeast-1:ami-0a75ff7b28897268c
    • Ap-southeast-2:ami-072b1b45245549586
    • Ca-central-1:ami-0e44086f0f518ad2d
    • Eu-central-1:ami-09cbcfe446101b4ea
    • Eu-west-1:ami-0d1cbc2cc3075510a
    • Eu-west-2:ami-0bd22dcdc30fa260b
    • Sa-east-1:ami-038596d5a4fc9893b
    • Us-east-1:ami-0843ca047684abe87
    • Us-east-2:ami-03d60a35576647f63
    • Us-west-1:ami-06f6efb13d9ccf93d
    • us-west-2:ami-0b5b8ad02f405a909
  • They are replacing:
    • Ap-northeast-1:ami-cbe000ad
    • Ap-northeast-2:ami-629b420c
    • Ap-south-1:ami-97bac2f8
    • Ap-southeast-1:ami-63b22100
    • Ap-southeast-2:ami-dd6c73be
    • Ca-central-1:ami-d02c93b4
    • Eu-central-1:ami-b243eedd
    • Eu-west-1:ami-61de3718
    • Eu-west-2:ami-904e5ff4
    • Sa-east-1:ami-c22057ae
    • Us-east-1:ami-05b6e77e
    • Us-east-2:ami-9b4161fe
    • Us-west-1:ami-efc9e08f
    • Us-west-2:ami-ce8c94b7

Bug fixes

• Fixed some bugs related to GitHub API response handling and webhook handling.
• Fixed issue with Scheduled Workflows when the services machine is restarted.
• Fixed changing the RabbitMQ hostname for Scheduled Workflows when externalizing.
• You can no longer create contexts with no names. If you are using a context with no names, it will no longer be accessible from your execution environment.
• We have optimized our handling of large amounts of build output and test results XML, to avoid out-of-memory errors.
• The CIRCLE_PULL_REQUEST environment variable was not being populated correctly in all cases when building across forks. This has been fixed.
• Fixed a bug where some commits with [ci skip] in the message were still being built.
• Fixed a bug causing Workflows to get stuck when infrastructure_failure happens after a job fails.
• Fixed a bug causing duplicate docker networks on same nomad client (if running build using machine:true AND vm-provider=on_host).
• Improved performance when using local storage. Previously, caching issues had been experienced when local storage was used rather than the default option of using S3 (selecting None under Storage Driver options from the Management Console).
• We have added more error checking and validation around Github’s API so the existing list commit endpoint no longer causes issues.
• Datadog API token field was stored in plaintext, now set as a password field.
• Fixed issue where workflows were constrained from fanning out to large number of jobs.

Fill out this form to receive updates about CircleCI through email.

What’s Fixed in Release 2.16.2

• Bug Fix: Fix a race condition that causes some threads to be stuck in certain use cases.

Fill out this form to receive updates about CircleCI through email.

What’s Fixed in Release 2.16.1

• Bug Fix: This release includes a new flag to disable GitHub cache warming. Note: Only add this flag if a high thread count is impacting the API service. To disable cache warming after completing the upgrade, complete the following steps:

1. Add the export DOMAIN_SERVICE_REFRESH_USERS=false flag to the /etc/circleconfig/api-service/customizations file on the Services machine.

2. Restart CircleCI by going to the Administrative console and clicking Stop Now, waiting for it to stop, then clicking Start.

• Bug Fix: Fixed issue preventing metrics from domain service from being forwarded.

• Security Fix: The SMTP password field in the dashboard was set to plain text. The password field will now be hidden.

Fill out this form to receive updates about CircleCI through email.

Note: It is currently a best practice to use a Services Machine with a minimum of 32GB of RAM. Starting in 2019 it will become required. See docs%20for%20our%20recommendation(s).

What’s New in Release 2.16

The following updates and fixes are also of note:

IMPORTANT We have upgraded Replicated to version 2.29.0 which requires Docker 17.12.1. Please follow the instructions provided in the Updating Server Replicated Version guide before upgrading.

New Feature We are excited to announce that you can now distribute your data and workload external to the Services Machine. The following services can be externalized; MongoDB, Redis, Nomad Server, RabbitMQ, Postgres and Vault. Please contact your CSM for the latest documentation.

New Feature Custom Metrics can now be accomplished via a Telegraf Output Configuration File. See the Custom Metrics section of the Monitoring section. Note: The Custom Metrics section in the admin UI been disabled in favor of a configuration file.

New Feature Users can now receive email notifications about Workflows

New Feature The PostgreSQL image has been updated to allow modifying the default configuration by creating the following file: /etc/circleconfig/postgres/extra.conf A list of configuration options can be found here.

Updated Removing EOL Banner on build emails

Updated VM Service stability improvements

Bug Fix Fix issue with contexts that would break after 32 days

Bug Fix Vault auth tokens will now renew periodically to prevent expiration

Bug Fix Fix for an issue where workflow listing pages would intermittently fail to render

Bug Fix We have fixed a small number of bugs affecting processing incoming web hooks from GitHub to CircleCI.

Bug Fix For installs that are using an HTTP or HTTPS proxy, the jobs will now ignore that proxy so that setup_remote_docker works.

Bug Fix Reduced the number of connections 1.0 builders make to the PostgreSQL database.

Security Fix Security fixes for potential cross-site scripting vulnerability and HTTP header injection vulnerability

Security Fix Forked PRs can no longer write the caches of parent projects by default for security reasons. It is possible to still write parent project caches from the fork if the “Pass secrets to builds from forked pull requests” (in Advanced settings) is enabled.

Update Improved metrics for VM provisioning for machine executor. This changed metric names, so if you are monitoring VM provisioning in your install already, you will need to reconfigure the monitoring dashboards for these new metrics.

• New metrics:
  • vm-service.gauges.available-vms and vm-service.gauges.running-vms replaced by vm-service.gauges.vms_by_status
  • vm-service.gauges.running-tasks replaced by vm-service.gauges.tasks_by_status
  • vm-service.gauges.oldest-unassigned-task replaced by vm-service.gauges.unassigned_tasks_age

Known Issue(s)

When a saved context is removed during the build graph phase it can cause a silent failure during job execution. \ In some instances the metrics section becomes hidden in the settings page. Please see this Knowledge article for instructions.

Fill out this form to receive updates about CircleCI through email.

Note: It is currently a best practice to use a Services Machine with a minimum of 32GB of RAM. Starting in 2019 it will become required. See docs%20for%20our%20recommendation(s).

What’s New in Release 2.15

This release contains an upgrade to MongoDB, service migrations for contexts and is a required upgrade. The following updates and fixes are also of note:

MongoDB Upgrade The MongoDB version running on Services VM will be automatically updated from 3.2 to 3.6 for you. If you have externalized MongoDB and wish to upgrade to 3.6, please contact your CSM

Bug Fix Restrict interaction of Workflows ‘Rerun’ dropdown to users who are part of GitHub organization for open-source projects.

Bug Fix Known issue for broken UI due to workflows with “.” in the name

Bug Fix Known performance issue with freezed browsers for organizations with many branches.

Bug Fix Known issue for Vault auth tokens not being automatically renewed. If you have externalized Vault, please contact your CSM before you upgrade to 2.15.0

New Feature: Splitting test files via Workflows. If your project has a large number of tests, you can use circleci tests split feature to reduce your job run time. For more information on how to setup tests split, please follow this document.

New Feature: Workflows UI will now show actors who have taken the following actions:

• Rerun workflows
• Approved jobs
• Canceled jobs

New Feature We are excited to announce more Metrics! Nomad job metrics can be enabled and will be emitted by the Nomad Server. Please see Nomad Metrics documentation for instructions on how to configure Nomad Clients. The following five types of metrics are reported:

• circle.nomad.server_agent.poll_failure: Returns 1 if the last poll of the Nomad agent failed, otherwise it returns 0.
• circle.nomad.server_agent.jobs.pending: Returns the total number of pending jobs across the cluster.
• circle.nomad.server_agent.jobs.running: Returns the total number of running jobs across the cluster.
• circle.nomad.server_agent.jobs.complete: Returns the total number of complete jobs across the cluster.
• circle.nomad.server_agent.jobs.dead: Returns the total number of dead jobs across the cluster.

New Feature Host (Services VM) and docker metrics can now be forwarded via Telegraf , a plugin-driven server agent for collecting & reporting metrics, to DataDog or a custom location. To enable and/or configure please go tohttps://example.com/settings#metrics. For more information please see the Monitoring documentation.

Performance Improvement Reduced the load time for all workflows pages.

Performance Improvement CircleCI has upgraded the mechanism that returns gettimeofday calls. As a result, heavy users of syscall will see an aggregate improvement in performance. Most noticeably this will improve the run time of projects with heavy usage of debugging/profiling tools and Xdebug with PHP.

Updated JVM_HEAP_SIZE is configurable for frontend, output-processor, and test-result containers. Please see JVM Heap Size Configuration documentation.

Updated For customers running the Static installation, Ubuntu 16.04 (Xenial) has been added as a supported OS. For additional details please see the docs.

Known Issue(s)

When a saved context is removed during the build graph phase it can cause a silent failure during job execution. \ When using the none storage driver, the cache will not be saved. This means that you currently cannot use caching in CircleCI when you are using the none storage driver.

This patch release fixes an issue introduced in 2.14.0. Please review the fix(es):

Fixed: Docker executor does not support configurable resource classes.

What’s New in Release 2.14

This release contains service migrations for contexts and is a required upgrade. The following updates and fixes are also of note:

Fixed: Pre-allocated VMs not being used. Customers reported that jobs using machine executor and setup_remote_docker were continuing to spin up (and terminate) new VMs at runtime instead of the pre-allocated pool.

Fixed: Unable to build in AWS region eu-west-2.

Updated: The title of the Builds page in the CircleCI application was changed to Jobs in several places to better reflect the structure of builds. Refer to the CircleCI blog post for details about the name change.

Updated: Contexts are now checked for unique naming across your installation of CircleCI instead of just within an organization. You will no longer be able to create duplicate context names within an account. Existing duplicates will be honored by the system, but should be changed to avoid future collisions. Please see the Contexts documentation for further information.

Updated: The test-results services cleans up the /tmp directory on restart and the JVM heap size increased from 256m to 512m.

Known Issue(s)

When a saved context is removed during the build graph phase it can cause a silent failure during job execution. \ When using the none storage driver, the cache will not be saved. This means that you currently cannot use caching in CircleCI when you are using the none storage driver.

Migration Tools: The 1.0 migration center was updated with additional content and helpful tools to migrate projects from CircleCI 1.0 to 2.0.

What’s New in Release 2.13*

• Fixed: Due to a certificate not being imported jobs were unable to submit after a container restart.
• Fixed: Stability issues within contexts service.
• Fixed: vm-service issue where AWS ECS and EBS volumes were left used. Added better idempotent client retries, API eventual consistency handling.
• Added: vm-service EBS Volumes cache evictor. Docker Layer Caching EBS volumes get deleted after 14 days of unused. You can control the inactivity period in Installation Admin Settings page.
• Added: Periodic checks to scale down any unused long running instances (e.g. due to reduction in prescaling configurations, or or due to a bug preventing termination of VMs in timely manner).
• Added: Optimizations to speed up provisioning. Resiliency to network/external changes.
• CircleCI is excited to announce that users on AWS can now forward host (Services VM) and docker metrics via Telegraf , a plugin-driven server agent for collecting & reporting metrics, to AWS CloudWatch. This is is the first step into bringing you greater observability into your installation. To enable and configure please go tohttps://example.com/settings#metrics. In this release the following metrics may be enabled: Docker , Networking , Disk , Memory, and CPU. For more information please see the Monitoring documentation.
• CircleCI is enhancing visibility by capturing usage statistics of CircleCI installations. This aggregate data will be used to better support you and ensure a smooth transition off 1.0. This optional feature is automatically available when you upgrade to the latest version of your installation or once a support bundle is downloaded. Below are the metrics being captured. If you would like access to this data in order to better understand your users please contact a CircleCI account representative. Please reference exhibit C within your terms of contract and our standard license agreement for our complete security and privacy disclosures. For more information please see the Usage Statistics documentation.

Weekly Account Usage

Weekly Job Activity

* this release contains migrations and is a required upgrade

The installable version of CircleCI v2.12.1 fixes an issue introduced in 2.12.0. GitHub cloud integrations were unable to create new contexts.

The installable version of CircleCI v2.12.0 fixes an issue in which an error occurs when uploading artifacts to S3 by updating AWS SDK for Java from v1.10.69 to v1.11.328.

This release fixes an issue with overwriting the known_hosts.local file. The fix allows keys to remain and enables the CircleCI upgrade process to append keys.

The installable version of CircleCI v2.10.0 fixes an issue in which jobs were unable to submit after a container restart due to the certificate not being imported.

The installable version of CircleCI Server 2.9.0 includes the following improvements and bug fixes:

• Resolution for a potential LDAP injection vulnerability
• Addition of support for the following LDAP fields:
  • Group Membership
  • Group Object Classes

See the Authentication document for the steps.

• Addition of default values for OpenLDAP and Active Directory
• Stability and bug fixes for an issue with test results
• Improvements to ready agent logging for support bundles
• Stability fixes and speed improvements for provisioning remote Docker engine
• Reliability improvements for the permissions service client
• Differentiation between the LDAP login and email to remove email as the assumed login value
• Fix for a bug that deselected all projects when a new user logged in for the first time
• Fix for a bug that was preventing Workflows from working correctly when located behind a proxy
• Bug fixes and improvements for Contexts

The installable version of CircleCI 2.8.0 includes the following fixes and improvements:

• An issue involving pre-allocated remote docker, machine VM creation, and other VM usage on AWS when the on-host VM option is selected has been fixed.

• An issue with the VM provider timing out when waiting for SSH to connect has been fixed. This upgrade supersedes any custom values for the Number of SSH attempts for starting VM setting and adds an equivalent field named Timeout for waiting for SSH to connect to VM(s) that you must set.

The installable version of CircleCI 2.7.0 includes the following fixes and improvements:

• Improvements to remote Docker instance provisioning and cleanup on AWS.

The installable version of CircleCI 2.6.0 includes the following fixes and improvements:

• It is now possible to create multiple Contexts in the Organization Settings of the CircleCI app. See the Contexts documentation for details and naming conventions.

The installable version of CircleCI 2.5.0 includes the following fixes and improvements:

• This release fixes a problem with DNS resolution failure on alpine-based containers. This problem was observed in the 2.3 release and was fixed for some service containers in 2.4. The 2.5 update fixes the issue for all service containers.
• This release includes improvements to scaling and garbage collection in the VM Service that should improve performance for installations with heavy use of machine executors or remote Docker.

The installable version of CircleCI 2.4.0 includes the following improvements and changes:

• Audit logging - This feature makes available a downloadable log of system events from the CircleCI Admin dashboard. This is the first release of audit logging; more events and improved data in the logs will be delivered in subsequent releases. Find detailed information about the data fields for each event in the Audit Logging section of the Security Features document.

• LDAP integration - It is possible to enable a new CircleCI installation to authenticate users with OpenLDAP or Active Directory credentials. New installations may be configured to have users authenticate through LDAP, including Active Directory support. Refer to the Authentication document for instructions.

• New build-agent - The new build-agent fixes some cases in which users were unable to log into a running job with SSH.

• VM Service improvements - Autoscaling of the VM Service has been improved, so machine executors should have more consistent performance at scale.

The installable version of CircleCI 2.3.0 includes the following improvements and changes:

• Users can now schedule workflows. The underlying cron-service has been added to the installation, though no specific configuration should be needed. See the documentation on Workflows Orchestration for details.
• Support for using private IP addresses in 1.0 builds is now enabled along with 2.0 builds
• A bug in the VM garbage collector was addressed that should help in situations when VMs were being leaked.
• We have disabled Replicated snapshots. Please see the documentation on backups if you have previously used Replicated snapshots.

The installable version of CircleCI 2.2.0 includes the following improvements and changes:

• New visualization for Workflows - When you click on a workflow in the CircleCI application, a diagram appears with the relationship of your jobs to one another and the complete sequence of the workflow.
• Updates to the onboarding flow - We now show a new user only projects already building on CircleCI rather than a list of all projects found in source control. This change was made to prevent inadvertent adding of projects to CircleCI during the onboarding process. We also added a confirmation dialog for any user opting to follow more than six projects and made it clearer how to add new projects not yet building on CircleCI.
• Support for object stores other than S3 - CircleCI 2.2.0 now supports the same object store options that are supported in 1.x. This is a prerequisite to future releases that will allow non-AWS installations of CircleCI 2.x.
• Updates to the VM Service; ACTION MAY BE REQUIRED IF YOU ARE USING AWS - The VM service has been updated to use volumes rather than persistent instances. This change allows operators to accommodate Docker Layer Caching with a smaller number of long-running VMs. If you installed using the suggested IAM permissions from prior versions you must update your IAM permissions with this upgrade. Refer to the policy template for the complete set of permissions.
• New Contexts feature; ACTION MAY BE REQUIRED IF YOU ARE USING THE HIGH AVAILABILITY CONFIGURATION - The Contexts feature uses a HashiCorp Vault secure storage mechanism. If your installation is running in HA mode, HashiCorp Vault needs to be externalized similar to the Mongo and Postgres databases. Refer to the Configuring High Availability documentation for instructions.

• Fix setting for certain proxy configurations.

• Added support for 2.0 builders to use object storage other than S3 (mirroring the options from the 1.x releases).
• Support for private images in docker jobs, including support for ECR authentication.
• New VM service and build agent to address certain situations that resulted in build infrastructure failures.
• Better accommodations for using proxies throughout the core infrastructure.

• Added support for proxies to the VM Service/Remote Docker setup.
• Fixed an issue in the mongo migration scripts when moving from 1.x to 2.x that caused some customers problems.
• Move to using Docker 17.03.2 (downgrade) to solve occasional freezes when starting/restarting.

• Fix an issue where users were unable to import large PostgreSQL data sets during an upgrade from 1.0 to 2.0

• Fix issue where /etc/hosts was not being applied properly to every container.
• Add ability to configure VM instance type in AWS EC2 when configuring the VM Service in the Management Console.

CircleCI 2.0 is a completely updated CI/CD platform that starts every run with a clean image which is automatically provisioned just-in-time for Linux jobs on the CircleCI application installed in your private cloud. The CircleCI 2.0 platform includes significant performance, stability, and reliability improvements along with first-class Docker support, Workflows (pipelines), Resource Allocation, and Contexts. The Management Console (Replicated) also includes new options for enabling 2.0 Builders and appropriate default settings.

If you have CircleCI installed you may access CircleCI 2.0 features on your current installation with no restrictions under your current agreement and support level. However, you must contact your CircleCI account representative for assistance with upgrading, to obtain migration scripts, and to receive a new license channel.

• CircleCI 2.0.0 is only available on AWS and may be installed with Terraform or manually.
• Teams may create a CircleCI 2.0 .circleci/config.yml file in their repositories to add new 2.0 projects incrementally while continuing to build 1.0 projects which use a circle.yml configuration file.

Documentation for the installable 2.0 application is available at CircleCI 2.0 Documentation for the following topics:

• Administrator’s Overview
• Administration FAQ
• Security Features
• Installing CircleCI 2.0 on Amazon Web Services with Terraform
• Configuring High Availability
• Administrative Variables, Monitoring, and Logging
• Introduction to Nomad Cluster Operation
• Backing up CircleCI Data

• Addresses an issue with CloudWatch metrics not being reported correctly.
• Address an issue with some build output being truncated in the UI.

• Addresses an issue with CloudWatch metrics not being reported correctly. UPDATE: some customers report this has not addressed their CloudWatch problems. The next release will have a new fix.

• Addresses an issue for installations pointing at github.com (rather than an instance of GitHub Enterprise) that prevented those installations from adding new projects.

• Fixes a bug that was causing lists of builds to be truncated in some circumstances.
• Fixes a bug where certain configurations of GitHub were not syncing email addresses properly.
• Fixes a bug where some users were seeing a beta UI that should have not have been showing up.
• Various minor UI fixes.

• Fixes a bug in new installs that prevented the first user from becoming an Admin.
• Fixes a bug with using the built-in Heroku deployment support.

• New Projects list in the Admin section gives you an overview of all projects across your entire installation of CircleCI Enterprise.
• New onboarding flow – new users will be prompted with a list of projects to follow drawn directly from their GitHub account.
• New option for users to choose “My builds” or “All builds” when viewing lists of builds.
• New JIRA integration for creating issues from a Build Page - configured under Project Settings.
• Option to externalize the underlying persistance storage (Postgres and Mongo) - talk to your CircleCI contact for details.
• Improved support for snapshot backup and restore via the Admin Console.
• Various performance improvements and minor bug fixes.

This is a security update to prevent logging temporary access tokens in build output. Those access tokens would have allowed a user to do a PUT request to overwrite an existing artifact only if all of the following were true:

1. They already had access to the build.
2. They identified the token in the build output.
3. The were to use the token within 30 minutes after the original artifact was uploaded.

A minor maintenance release, primarily to improve the single-box installation process for new trials of CircleCI Enterprise. Also, we have now follow advice from GitHub to not check if emails in GitHub Enterprise are “verified” — we will continue to check verified status if you are using github.com. If you have reason to change “verified” checks on your users’ emails let us know.

Note for Admins

This release requires extra steps than usual. Please see <(/ja/docs/enterprise/upgrade-for-1.46.x-and-earlier/)> for detailed instructions.

We have released patches to our AMIs and other infrastructure to address CVE-2016-8655. We recommend all CircleCI Enterprise installations follow the instructions below to update both their Services box and their Builder fleet.

If you have any questions or difficulties please contact enterprise-support@circleci.com.

Update the Services box:

1. As always, ensure your data is backed up.
2. Shut down CircleCI in the Replicated console (or via the CLI).
3. Update the kernel using the provided install_kernel_master function below.
4. Restart the machine.

#!/bin/bash
function install_kernel_master() {
    echo '>>> Installing Kernel'
    apt-get update
    apt-get install linux-image-3.13.0-105-generic linux-headers-3.13.0-105-generic linux-image-extra-3.13.0-105-generic
    apt-cache policy linux-image-3.13.0-105-generic linux-headers-3.13.0-105-generic linux-image-extra-3.13.0-105-generic
}

Update the Builder fleet:

For the builder fleet, update the Launch Configuration to use the updated AMI from the list below:

• ap-northeast-1 = “ami-07f09d60”
• ap-northeast-2 = “ami-90588ffe”
• ap-southeast-1 = “ami-4f54f82c”
• ap-southeast-2 = “ami-040c3467”
• eu-central-1 = “ami-3465a35b”
• eu-west-1 = “ami-d70421a4”
• sa-east-1 = “ami-2e6bf242”
• us-east-1 = “ami-e68f89f1”
• us-west-1 = “ami-901c4af0”
• us-west-2 = “ami-0c57fc6c”

If you are using our Terraform scripts, you can download the new script https://github.com/circleci/enterprise-setup/blob/master/circleci.tf and run terraform apply. We’ve already updated the scripts to include the new AMIs, so terraform should launch new builders automatically with the patched version, and cycle your fleet.

If you are using a non-AWS environment, use the same method to patch your builders you used to patch the Services box.

If any of the above does not apply to your environment, or you encounter issues with your upgrades please contact: enterprise-support@circleci.com.

• Fix for a bug causing builders to not honor CIRCLE_CONTAINERS_SUBNET when passed in the initialization.
• Builds with high parallelism were experiencing slow performance on the build detail page in the web UI – this should now be improved.
• Some emails generated by CircleCI had hard-coded links to circleci.com. These should now point at your installation host.
• A handful of minor UI fixes.

We have released patches to our AMIs and other infrastructure to address CVE-2016-5195. We recommend all CircleCI Enterprise installations follow the instructions below to update both their Services box and their Builder fleet.

If you have any questions or difficulties please contact enterprise-support@circleci.com.

Update the Services box:

1. As always, ensure your data is backed up.
2. Shut down CircleCI in the Replicated console (or via the CLI).
3. Update the kernel using the provided install_kernel_master function below.
4. Restart the machine.

#!/bin/bash
function install_kernel_master() {
    echo '>>> Installing Kernel'
    apt-get update
    apt-get install linux-image-3.13.0-100-generic linux-headers-3.13.0-100-generic linux-image-extra-3.13.0-100-generic
    apt-cache policy linux-image-3.13.0-100-generic linux-headers-3.13.0-100-generic linux-image-extra-3.13.0-100-generic
}

Update the Builder fleet:

For the builder fleet, update the Launch Configuration to use the updated AMI from the list below:

• ap-northeast-1 = “ami-59389e38”
• ap-northeast-2 = “ami-a2e236cc”
• ap-southeast-1 = “ami-a80fa9cb”
• ap-southeast-2 = “ami-53241930”
• eu-central-1 = “ami-5a59a035”
• eu-west-1 = “ami-e0c78993”
• sa-east-1 = “ami-3832af54”
• us-east-1 = “ami-d0396bc7”
• us-west-1 = “ami-76226916”
• us-west-2 = “ami-938420f3”

NOTE: The following AMIs are not maintained by CircleCI, but contain a patched version of stock ubuntu.

• gov-west-1 = “ami-34df6755”
• cn-north-1 = “ami-92f622ff”

If you are using our Terraform scripts, you can download the new script https://github.com/circleci/enterprise-setup/blob/master/circleci.tf and run terraform apply. We’ve already updated the scripts to include the new AMIs, so terraform should launch new builders automatically with the patched version, and cycle your fleet.

If you are using a non-AWS environment, use the same method to patch your builders you used to patch the Services box.

If any of the above does not apply to your environment, or you encounter issues with your upgrades please contact: enterprise-support@circleci.com.

• Fix for a bug that stopped certain CloudWatch metrics from working.

• Fix for a bug that caused UI problems for certain Pull Requests

• Minor changes to help debug some esoteric issues.

• Fixed layout problems in Safari on Mac.
• Fixed issue with tag builds interacting with the Auto-Cancel of Redundant Builds setting.
• Fixed performance of the build page, including awkward interaction when expanding build steps
• Fixed environment variables not sorting properly in project settings

• Note for Admins: we will now block your upgrade if there are pending migrations that need to be performed against your databases.
• We have changed the default behavior for builds with parallelism set above the current total container count in your build fleet. Previously we would auto-cancel any build with a parallelism above the total number of containers in your fleet. Those builds will now remain in the queue until the fleet is big enough to accommodate their parallelism setting. Builds behind those builds will continue to be bumped ahead, however, so the behavior is somewhat different than normal queuing behaviors, which is a strict FIFO model.
• New System Settings option to set the maximum parallelism for all projects. This allows you to prevent teams from using more of your fleet than intended.
• New option to auto-cancel “redundant” builds on a branch - found in the project settings, this new option allows you to set builds on branches other than your default branch to auto-cancel any builds on that same branch already running. This avoids running multiple simultaneous builds on the same working branch if you push several times in quick succession.
• On the Systems Settings page under Admin you can now enter “Container Tweaks” that will be run whenever containers are started in your build fleet. This feature allows you to make adjustments to the build environment prior to builds running, so you can save time in each build. Note that these tweaks are only appropriate for things you want to apply to the environment for ALL of your builds.
• We now parse your circle.yml before the build runs and fail if there are errors. Before, the build would start and errors would be surfaced only once they caused a failure in a running build.
• Fixed a bug preventing artifacts from being generated in some circumstances after a failed build.
• The Artifacts API now allows cross-origin requests when requesting specific artifacts. In the past you could do this when using the API to get a list of your artifacts but not for downloading specific artifacts. You can now do both with the API.
• Fixed a bug that prevented some cases of triggering builds via the API. If you have had problems triggering builds with the API you may need to recycle your build fleet for this fix to take effect.
• Fixed a bug that would cause problems if someone’s access was removed from a project in GitHub Enterprise but that person’s token was being used by CircleCI Enterprise. Other users should now be able to continue using the project as expected if the original token’s owner no longer has access.
• Fixed a bug that prevented builds cancelled from the UI from actually cancelling in a timely fashion.
• Various design improvements such as better notifications and confirmation dialogs when changing various settings, new links on the build page and builds list to make it easier to link directly to specific containers or parts of your build output, better information for builds marked as “Not Run”, hiding some settings that aren’t applicable to Enterprise customers, and various minor improvements and bug fixes.

• Fix for an issue that was causing builds to fail that were triggered by users with a github account but no circleci account.

Note: we are shifting our versioning to better reflect the current release semantically.

• PR-only builds - We have added functionality to only run builds when a pull request is open. To enable this functionality you can navigate to “Advanced Settings” for your project and enable the “Only build pull requests” option.
• Chrome notifications - see the docs
• Fixed a bug where builds triggered by people who don’t have seats in CircleCI Enterprise were not running. They now should run as before.
• Some UI changes to make updating data in the web interface more consistent.
• Reliability improvements by changing retry logic in various places.

Bug fix: We’ve fixed a bug that “–branch” option was missing when “–single-branch” is used.

New Feature: The maximum size of files that you can upload for caching during builds has been fixed to 5G. The size is now bumped 20G by default on CircleCI Enterprise and customers can also override the default value to even larger size.

To override the default value, you need to run the following in a REPL of Service Box.

(settings/set {:max-file-cache-bytes <max-size-by-bytes>})

To unset the overridden value and use default, run the following.

(settings/unset! [:max-file-cache-bytes])

New Feature: Custom base URL for version control webhooks. When a new project is added, CircleCI will add a webhook to the GitHub repository of the project. With this new feature, you can override the default webhook base URL from the System Settings page under the Admin tools (available to designated administrative users). This feature is useful when your instance is behind firewall or other proxy and cannot directly receive webhooks from GitHub.

This release fixes a bug in the URL structure used to serve build artifacts.

The artifact URL format was recently changed to handle security concerns related to CircleCI’s hosted offering. The security concerns do not affect CircleCI Enterprise customers, but the change caused issues fetching build artifacts in CircleCI Enterprise installations. This release reverts the CircleCI Enterprise artifact format and resolves the issue.

• Please Note: If you are using OS X builds you will need to run a manual migration as part of this release. After upgrading you will need to run this in a REPL: (circle.backend.model.esxi-vm/run-migrations!). Please talk with your account manager if you need further instructions.
• The Admin Users page now has links to the builds of each user.
• The API has a new endpoint /api/v1/admin/licensing that returns information about the number of seats available, number being used, when the license will expire, and how many inactive users there are in the system.
• Fixed a bug where links to the documentation broke.
• Various small bug fixes and UI improvements.

Please make sure that you are using the latest provision-builder.sh and init-builder-0.2.sh before you upgrade to this release:

https://s3.amazonaws.com/circleci-enterprise/provision-builder.sh

https://s3.amazonaws.com/circleci-enterprise/init-builder-0.2.sh

As part of this release we’re changing the behavior of artifacts to only serve an allowed set of content-types. This means we won’t serve .html files as text/html. This is a security risk on CircleCI Enterprise since artifacts are served on the same domain as the rest of the site – as a result, any user or malicious code used as part of your build can push a specially-crafted artifact and gain control of another user’s account.

If this is an issue, you can override this behavior by setting “Serve artifacts with unsafe content-types” in the admin console. We don’t recommend this, but we’re providing it for backwards compatibility.

This release also includes some changes to container networking. Containers now each use a /24 in the subnet 172.16.1.0/16 by default.

If this conflicts with your private network, or if you were editing lxc-net manually in order to fix a prior conflict, you can now use CIRCLE_CONTAINERS_SUBNET and CIRCLE_CONTAINERS_SUBNET_NETMASK_LENGTH on the builders to configure those. See “Adjusting Builder Networking” in the configuration section.