Before Upgrading

See the CircleCI server 4.7 release notes and upgrade guide for this release.

NOTE: Vault is being deprecated and will no longer be supported in server 5.0. Refer to our script for steps to migrate to Tink.

What’s New in Release 4.7.0

The v4.7 release introduces security improvements with the implementation of rootless containers.

NEW FEATURES

• Approval jobs can now be canceled through the CircleCI UI or via the API.
• Windows container runner preview

CHANGES

• Server components now run as rootless containers, enhancing security.
• The Nomad server ReplicaSet is now scaled to 5 pods by default, which improves execution stability at scale.
• Nomad clients require IPv6 support within the kernel.
• Configuration for the RabbitMQ PersistantVolumeClaim (PVC) is now exposed through server Helm values. For more details, see the docs.

BUG FIXES

• Resolved an issue where frontend pods would not automatically detect and apply a new server license.
• Fixed a bug where a workflow could be prematurely marked as failed before all non-blocking jobs were run.
• Fixed a configuration issue that could cause connection refusals between Kong and Soketi following an upgrade to version 1 of Soketi.
• Addressed a typo in the Helm values for machine_provisioner.machine_agent_base_url. The correct template value should be machine_provisioner.agent_base_url.

SERVICE CHANGES

Deprecated components removed with this release:

• web-ui-404: Previously served the 404 error page. Its functionality has now been merged into the main web-ui component.
• Support for GitHub Enterprise versions <= 2.2: Code supporting these versions has been removed, as they are no longer supported by GitHub.

DATABASE MIGRATIONS

The following databases will run migrations when upgrading to this version:

• authenticationservice
• conductor_production

KNOWN ISSUES

• SSH reruns in air-gap will time out, leaving the job in an error state
• Vault may not refresh its client token after a month of uptime. Migrate to Tink to resolve this issue.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• Machine builds using instances with IMDSv1 enabled is not supported

To learn more about Server 4.7 installation, migration, or operations please review our documentation.

IP Ranges now support parameters providing more dynamic control of when ip ranges is used in builds.

Example

version: 2.1

parameters:
  ip_ranges:
    type: boolean
    default: true

executors:
  base:
    docker:
      - image: cimg/base:stable

jobs:
  job_a:
    executor: base
    circleci_ip_ranges: << pipeline.parameters.ip_ranges >>
    steps:
      - run: echo "Running job a"

workflows:
  workflow-a:
    jobs:
      - job_a

As a continuation of CircleCI’s update to enable access to “GitHub App functionality” to all CircleCI users (including users of the GitHub OAuth App integration), the “Triggers” tab in Project Settings of the CircleCI web app now shows a “GitHub OAuth App” trigger and a badge to indicate the integration type. This is intended to make it easier for all users to understand what is currently set up within their CircleCI project.

As part of CircleCI’s update to enable access to “GitHub App functionality” to all CircleCI users (including users of the GitHub OAuth App integration), the “Pipelines” tab in Project Settings of the CircleCI web app now shows a “GitHub OAuth App” or “Bitbucket Cloud” pipeline and a badge to indicate the integration type. This is intended to make it easier for all users to understand what is currently set up within their CircleCI project. A subsequent update will do the same for the “Triggers” tab.

Known issues: The organization name will be removed from the “Config Source” column to only show the repository name.

A new v2 API endpoint for triggering pipelines via API is now available to all organizations that integrate through GitHub (GitHub OAuth and GitHub App) and Bitbucket (Cloud and Data Center). This functionality is not yet available to organizations that integrate through GitLab.

The endpoint is:

POST /api/v2/project/{provider}/{organization}/{project}/pipeline/run

This is now the recommended API to be used to trigger pipelines.

The pre-existing endpoint for triggering new pipelines - which is only available to organizations integrating through GitHub oAuth or Bitbucket Cloud - continues to be supported.

Historically, CircleCI users have been limited to setting up validation systems that only worked within the scope of a single repository.

If a pipeline’s YML config was stored in Repo A, that pipeline could only be triggered from events on Repo A (or from a Custom Webhook). This has left many customers unable to set up the validation system that they want.

We have launched additional enhancements to pipeline and trigger configuration, which will enable customers to build any validation setup that they need.

Read more on our community forum.

Job Filters now support expression based conditions, much like contexts. This enables you to conditionally run jobs based on pipeline values. This supports you optimizing pipelines to lower costs, decrease time to feedback, or run specific jobs based on context of the source of change. Previously filtering was extremely limited to branches and evaluating to only and ignore.

Example for a Deploy workflow:

workflows:
  deploy:
    jobs:
      - init-service
      - build-service-image:
          requires:
            - init-service
      - dry-run-service:
          requires:
            - init-service
          filters: pipeline.git.branch != "main" and pipeline.git.branch != "canary"
      - publish-service:
          requires:
            - build-service-image
            - test-service
          filters: pipeline.git.branch == "main" or pipeline.git.tag starts-with "release"
      - deploy-service:
          context:
            - org-global
          requires:
            - publish-service
          filters: pipeline.git.branch == "main" and pipeline.git.commit.subject starts-with "DEPLOY:"

We will be following on this with expression based evaluations for When Statements which enable more flexibility in how you choose when steps or workflows run. When Statements are also currently limited in their evaluation options. Improving When Statements will expand your options optimizing for cost and speed.

We are enabling users who are a part an organization that integrates with CircleCI’s GitHub OAuth App to use functionality that was previously only available to organizations that integrate with CircleCI’s GitHub App.

The immediate change that users will see is the presence of “Pipelines” & “Triggers” tabs in Project Settings of the CircleCI web app.

Read our community forum for details.

GitHub App users who granted their App access to more than 100 repositories have been running into issues when creating Pipelines and Triggers on CircleCI, because the repository dropdown would only display a limited number of repositories.

We have now replaced the dropdown inputs with a search component, which lets users search repositories by keyword.

Note: at the moment the search results include all of an organization’s public repositories. We plan to adjust this behaviour in the near future.

Until today, the Pipeline property “Fallback branch” determined which branch would be used to fetch config and code for pipelines triggered via Custom Webhook. This field could only be configured by editing the Pipeline associated with a Custom Webhook, rather than by editing the Custom Webhook itself, and was poorly discoverable.

Now, the field “Fallback branch” has been removed from Pipeline configuration, and two new fields have been added to Custom Webhook setup:

• Config branch, which determines the branch to be used to fetch config
• Checkout branch, which determines the branch to be used to check out code when running a checkout step.

For existing custom webhook triggers, both new fields have been populated with the “fallback branch” value of the associated pipeline, so that existing behaviour is preserved.

Pipelines connected via GitHub App or Bitbucket Data Center can now be run via API, using a new API V2 endpoint:

https://circleci.com/api/v2/project/circleci/<org-id>/<project-id>/pipeline/run

See the docs for usage instructions. The new endpoint is not yet documented in the V2 API docs.

Orbs are the leading method to abstract away shared aspects of your pipelines. They make it possible to simplify the complex, share, and maintain jobs and commands across your organization. For large organizations, private orbs cab provide efficiency and scale.

Orb security settings now allow organizations to enable only private orbs. Previously orb settings required enabling uncertified orbs in order to enable private orbs. This prevented some organizations, whom were concerned about uncertified orb security, from benefiting from the private orbs.

For more information on orbs and creating your own private orbs see our documentation.

With CircleCI’s Evals orb 1.x.x, users were able to configure CircleCI to orchestrate evaluations of their LLM-enabled applications within their CI pipeline.

Evals orb 2.0.0 builds upon version 1.x.x, introducing Evaluation Testing. With Evals orb 2.0.0, users are able to automate the entire process of running evaluations, reviewing evaluation results and determining whether results meet application performance criteria. This removes the need to manually trigger evaluations and manually review evaluation results. Automating evaluations with CircleCI enables developers to automatically determine whether or not to merge proposed changes, or deploy their AI application.

To use Evaluation Testing, users configure test cases to determine if the results of an evaluation meet specified conditions. Each test case represents a scenario with an assertion to check if a specified condition is true or false. If any test case fails, the evaluation test fails. You can configure test case assertions such as Thresholds (Check whether 1 or more results fall within a range), and Equality (Check whether results provide an expected answer)

Evaluation Testing results are presented in the CircleCI web UI in two locations:

1. In the step details
2. In the tests tab

Runner Release 3.0.25

Both Machine and Container Runner:

• Sign docker manifests to be verified with the CircleCI Public Key
• Security fixes
• Increase time between deleting existing task-agents to prevent unexpected removals during task execution

Machine Runner:

• Set KillMode=process for linux based machine runners

We have added release markers and release jobs to CircleCI Releases!

Release markers provide a lightweight way for users to generate a changelog and easily track their releases without requiring a full CircleCI Releases setup (no agent install needed!). Release markers can be configured for releases to any deployment target (ie. ECS, EC2, VMs, k8s, etc). See our documentation for information on setting up release markers.

Release jobs replace kubernetes manifest annotations as the way to configure deployment triggers. Release jobs also allow users to monitor their kubernetes release process as part of the overall deployment workflow in CircleCI, making it easier to find releases for a given pipeline. Users can also use release jobs to orchestrate releases across multiple environments (for example, deploy to production only after a successful release in a staging environment). See our documentation for information on setting up release jobs.

We’ve added additional meta data using the “tag” key to audit logs to help customers identify how workflow reruns were triggered.

Workflow tags include:

• setup
• rerun-single-job
• rerun-workflow
• rerun-workflow-from-beginning
• rerun-workflow-from-failed
• rerun-workflow-from-job-ids-sparse-tree
• rerun-single-job-with-ssh

Example audit log entries

Workflow Started:

workflow.start
{"job":{},
 "workflow":{"id":"247...","name":"test-and-build","tag":null}}

workflow.job.context.request
{"job":{"context_uuids":[],"contexts":[],"id":"408...","job_name":"test","job_status":"pending"},
 "workflow":{"id":"247...","name":"test-and-build","tag":null}}

workflow.job.start
{"job":{"contexts":[],"id":"408...","job_name":"test","job_status":"running"},
 "workflow":{"id":"247...","name":"test-and-build","tag":null}}

workflow.job.finish
{"job":{"contexts":[],"id":"408...","job_name":"test","job_status":"failed"},
 "workflow":{"id":"247...","name":"test-and-build","tag":null}}

Retry the original workflow with rerun job with SSH enabled:

workflow.retry
{"job":{},
 "workflow":{"id":"247...","name":"test-and-build","tag":null}}

SSH enabled rerun started:

workflow.start
{"job":{},
"workflow":{"id":"811...","name":"test-and-build","tag":"rerun-single-job-with-ssh"}}

workflow.job.context.request
{"job":{"context_uuids":[],"contexts":[],"id":"c6a...","job_name":"test","job_status":"pending"},
 "workflow":{"id":"811...","name":"test-and-build","tag":"rerun-single-job-with-ssh"}}

workflow.job.start
{"job":{"contexts":[],"id":"c6a...","job_name":"test","job_status":"running"},
 "workflow":{"id":"811...","name":"test-and-build","tag":"rerun-single-job-with-ssh"}}

workflow.job.finish
{"job":{"contexts":[],"id":"c6a...","job_name":"test","job_status":"canceled"},
 "workflow":{"id":"811...","name":"test-and-build","tag":"rerun-single-job-with-ssh"}}

The pipelines page can load slowly for some users. The number of pipelines and number of projects can contribute to that slowness. A recent change to one of the underlying data services has been made resulting in a 30%-40% improvement in performance.

We continue to work on improving the pipelines page experience. Our current focus is on load time. We do have plans to introduce new filter and search capabilities in the future.

We have added an API and UI component to approval jobs that allows you to “cancel” the approval job. Previously it was not possible to cancel approval jobs. This limitation forced you to either approve the job or let it expire, which takes 90 days. This led to a suboptimal experience with jobs left in a non-terminal state. This could impact performance metrics, as well pipelines views. It also led to non-intuitive actions being taken like cancelling the workflow which cancels all remaining jobs.

We are following up this feature with more flexible require statements. So you can have a job depend on an approval while another depends on cancellation. This is great for cleanup jobs, notifications, or other tasks that may only need to happen or not happen upon cancellation.

Previously, when processing a configuration file, CircleCI would process all workflows defined in the workflow. Upon completion of processing, errors such as missing or undefined parameters would be included in feedback.

CircleCI will no longer process workflows defined in a configuration file that aren’t going to be run. This means errors that might exist within that workflow will also not be caught and reported back to the user.

During development or troubleshooting, users should ensure a workflow has run, and it’s free of any errors. This can be done by making sure any conditionals like when: <complex condition> are replaced with when: true.

As previously reported https://circleci.com/changelog/launch-agent-1-x-variants-eol/.

Support for all 1.x variants of runner launch agent has ended. It has been replaced with Machine Runner 3.0. Any remaining access will be removed September 17th, 2024. For those that haven’t migrated, your builds on runner will fail.

You can learn more about machine runner and migration here.