With CircleCI’s Evals orb 1.x.x, users were able to configure CircleCI to orchestrate evaluations of their LLM-enabled applications within their CI pipeline.

Evals orb 2.0.0 builds upon version 1.x.x, introducing Evaluation Testing. With Evals orb 2.0.0, users are able to automate the entire process of running evaluations, reviewing evaluation results and determining whether results meet application performance criteria. This removes the need to manually trigger evaluations and manually review evaluation results. Automating evaluations with CircleCI enables developers to automatically determine whether or not to merge proposed changes, or deploy their AI application.

To use Evaluation Testing, users configure test cases to determine if the results of an evaluation meet specified conditions. Each test case represents a scenario with an assertion to check if a specified condition is true or false. If any test case fails, the evaluation test fails. You can configure test case assertions such as Thresholds (Check whether 1 or more results fall within a range), and Equality (Check whether results provide an expected answer)

Evaluation Testing results are presented in the CircleCI web UI in two locations:

1. In the step details
2. In the tests tab

We have added release markers and release jobs to CircleCI Releases!

Release markers provide a lightweight way for users to generate a changelog and easily track their releases without requiring a full CircleCI Releases setup (no agent install needed!). Release markers can be configured for releases to any deployment target (ie. ECS, EC2, VMs, k8s, etc). See our documentation for information on setting up release markers.

Release jobs replace kubernetes manifest annotations as the way to configure deployment triggers. Release jobs also allow users to monitor their kubernetes release process as part of the overall deployment workflow in CircleCI, making it easier to find releases for a given pipeline. Users can also use release jobs to orchestrate releases across multiple environments (for example, deploy to production only after a successful release in a staging environment). See our documentation for information on setting up release jobs.

Before Upgrading

See the CircleCI server 4.6 release notes and upgrade guide for this release.

NOTE: Vault is being deprecated and will no longer be supported in server 5.0. Refer to our script for steps to migrate to Tink.

What’s New in Release 4.6.0

The v4.6 release introduces various UI refreshments and improvements.

NEW FEATURES

• Several enhancements to the CircleCI web app, including:
  • A new “User Homepage”.
  • Updated styling for the left-hand navigation bar.
  • Top navigation bar that allows easy access to user-level actions.
• Pipelines are now labeled whenever they are rerun from start or fail, or with SSH.
• Project-based context restrictions. For more information see the docs.

CHANGES

• Old, expired workflows are now automatically canceled after 90 days.
• The pipeline.in_setup pipeline parameter has been removed.
• Kong, which is used for server API management, has been upgraded from 2.x to 3.x.

BUG FIXES

• Fixed a bug that prevented jobs from being rerun with SSH.
• Resolved an issue that prevented project loading on the Insights dashboard.
• Addressed regressions from server 4.2:
  • Introduced a configuration option at the organization level to set the default resource class for a Docker job. In server 4.3, this was previously hardcoded to large.
  • Added support for absolute paths in defining a GCP network under the machine provisioner configuration. This feature is particularly useful for VM jobs using shared networks, enabling explicit network selection. This corrects a regression from the previous VM service architecture.

NEW SERVICES

New services introduced with this release:

• authentication-service - provides an interface for user and identity management in the CircleCI system

DATABASE MIGRATIONS

The following databases will run migrations when upgrading to this version:

• builds_service
• conductor_production
• permissions

KNOWN ISSUES

• SSH reruns in air-gap will time out, leaving the job in an error state
• Vault may not refresh its client token after a month of uptime. Migrate to Tink to resolve this issue.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.6 installation, migration, or operations please review our documentation.

m1 medium is now the default resource class for Performance and Scale Plan customers. This applies only to jobs that do not have a resource class specified.

Update jobs using gen2 medium by June 28 to avoid failing builds. Learn more.

The official CircleCI Evals Orb makes it easy to integrate LLM evaluations into a CI pipeline, and to review evaluation results without context switching. The output of evaluations run through the Evals Orb is stored in CircleCI, and is accessible as a job artifact and as a PR comment added automatically by CircleCI.

Currently, the Evals Orb exposes commands to run evaluations through two popular LLMOps tools: LangSmith and Braintrust. If your evals leverage a different tool, let us know at ai-feedback@circleci.com. You can also contribute directly to the official Orb, by opening a PR on the public repository.

More resources on evaluating LLM-enabled applications are available in our documentation.

Get access to all your CircleCI usage data via our Usage API.

Query your usage data for any 32-day window (or less) going back up to 13 months. Data is refreshed daily through the previous full day. Understand your pipelines, workflows, and jobs, your credit consumption and RAM/CPU utilization on the job run and much more.

Hit the API to query your data, get back a link from which you can download a CSV, then load it into a data warehouse of your choice for deep CircleCI usage analysis.

For more information, check out our documentation.

Expression based context restrictions enable developers to restrict contexts based on pipeline values such as branch, ssh rerun and other attributes of the pipeline. This provides greater control over context access for things like contexts only available to production branches, builds run by specific actors and more. For security concerned and complex workflows customers are able to limit access to contexts to meet their specific security needs.

General documentation and API documentation are available to learn more.

With the launch of our release agent 1.0.0, CircleCI releases is now generally available. Installing the release agent in your kubernetes cluster gives users org-wide visibility into all components and releases via the Releases UI.

Our agent also integrates with Argo Rollouts to give customers more visibility and control over their progressive releases and use analysis runs to fully automate their release process based on observability metrics. Check out our docs to learn more!

CircleCI’s Visual Studio Code extension now lets customers trigger pipelines from VS Code based on local, unversioned configuration files. Users can choose to run a full pipeline, or select only the individual jobs and workflows they want to run by using checkboxes. This will shorten the feedback cycle and enable developers to iterate fast on their configuration, without having to commit their trial and error changes to the VCS.

This feature is controlled by an org-level opt-in setting, and an opt-out project level setting.

Check out the feature docs for more information.

Container Runner SSH Rerun (Preview)

Container runner now supports SSH Rerun. Visit our documentation on setup and usage.

For more information on SSH rerun please see the following documentation.

Users can now orchestrate their model deployments across multiple environments through the Amazon SageMaker Orb. They can also track and manage their deployments from within the Releases UI.

See our documentation for more information.

Inbound Webhooks are now available in open preview.

With this new feature, you can trigger a pipeline from anywhere that emits a webhook or run a curl command. With this change, we also updated pages such as the Dashboard to accurately reflect that pipelines can now come from outside your VCS.

Only available to users authenticated through our GitHub App integration. See our community forum and an AI use case for more.

Now in preview, three new resource classes that use GPUs:

1. gpu.nvidia.small.gen2: uses 1 NVIDIA Tesla A10G GPU (4 CPU / 16 GB RAM 200 credits/min)
2. gpu.nvidia.small.multi: uses 2 NVIDIA Tesla T4 GPUs (4 CPU / 16 GB RAM 160 credits/min)
3. gpu.nvidia.medium.multi: uses 4 NVIDIA Tesla T4 GPUs (8 CPU / 30 GB RAM / 240 credits/min)

Only available to users on our Scale pricing plan. See our community forum for more.

Upon encountering a failed step, users now can solicit guidance at the press of a button. This takes away the need to copy and paste the error message into your favorite search engine to find out what the problem might be and how to fix it.

Admins can opt out of this experiment by navigating to the Advanced section of Organization Settings.

The feature is powered, currently, by OpenAI. The only data sent to OpenAI is the error message in the output from the failed step. Before the message is sent any sensitive information such as passwords or access keys is removed. None of the data provided is being used to train the model.

Let us know what you think in the Discuss thread.

What’s New

CircleCI now offers Apple Silicon M1 support on our macOS executor. Customers on annual plans can take advantage of the improved performance by updating their config to resource_class: macos.m1.large.gen1.

Learn more by reading our announcement and documentation.

What’s New

Visual Studio Code users can now re-run CircleCI jobs with SSH, through either the integrated terminal, or a dedicated remote development window. You can automatically open an SSH connection within the terminal, or use a remote window to work on a CircleCI remote machine as if it was a VS Code workspace, which allows you to browse and view files easily, as well as take advantage of supported extensions.

For more details, see the Discuss post (including a quick demo), or the docs on VS Code Marketplace.

What’s New

Fail fast reduces wasted time and helps you optimize credit usage.

Configure your testing jobs to stop executing shortly after finding the first test failure, even in jobs where parallelism is enabled.

You can read more about the fail fast open preview in the Discuss post, and in the documentation.

What’s New

The CircleCI VS Code extension is now generally available. This extension provides an interface to visualize and manage CircleCI pipelines directly from VS Code. It also offers contextual help when creating, modifying, and editing CircleCI YAML configuration files.

By making existing CircleCI features available in VS Code and facilitating configuration file editing, the extension reduces context switching for CircleCI users (i.e. between the code editor and the browser).

Head to the VS Code Marketplace to view the full list of capabilities and download the extension for free.

What’s New

We have published the official CircleCI Visual Studio Code extension. This extension provides an interface to visualize and manage CircleCI pipelines directly from VS Code. It also offers contextual help when creating, modifying, and editing CircleCI YAML configuration files.

By making existing CircleCI features available in VS Code and facilitating configuration file editing, the extension reduces context switching for CircleCI users (i.e. between the code editor and the browser).

Head to the VS Code Marketplace to view the full list of capabilities and download the extension for free. The CircleCI VS Code extension is currently in open preview.

What’s New

Container runner, a more scalable and container-friendly self-hosted runner, has moved out of the open preview stage and is now in general availability.

Find more information about the release of container runner in the Discuss post. Read more about container runner in CircleCI’s documentation.

What’s New

Project roles are a new feature that allows customers to limit organization-level access, while giving individuals or teams privileged access to specific projects. This feature is currently available for projects on GitLab SaaS support.

For more details on available roles and their permissions, refer to our GitLab SaaS integration documentation.

What’s New

Container agent is a new type of self-hosted runner in open preview. Container agent adds support for the Docker executor with self-hosted runners, as well as basic auto-scaling and integration with Kubernetes.

Read more in the Discuss post.

Refer to our documentation for more information.

CircleCI server v4.0.0 is now generally available. The feature set for this release are equal to server v3.x. The installation and upgrade processes for v4.x have changed significantly.

Server v4.x offers greatly improved security handling, installation, and update processes. Server v4.x is installed using helm charts and images that can be pulled ahead of time to comply with your security processes. Installation processes can also use artifacts pulled from customer-managed registries. Server v4.x makes improved use of Kubernetes secrets and removes the requirement to grant permissions and network access to third-party tools.

Before upgrading

See the CircleCI server 4.x release notes for upgrade notes for this release.

What’s new in release 4.0.0

New features

• Two new Arm resource classes are now available xlarge and 2xlarge. For more information on using Arm, see the Arm docs.
• CircleCI server v4.x images and helm charts are available for licenced users.

Fixes

• Fixed a bug that was causing a huge quantity of error logs in the distributer-internal pod.
• Prometheus is no longer included by default in CircleCI server v4.x.
• The default concurrency limit has been increased to avoid it being reached.
• Docker layer caching volumes are now deleted after 3 days if unused. Previously they were deleted after 7 days.
• kong is now placed behind NGINX to simplify loadbalancer setup.
• The loadbalancer has been renamed from circleci-traefik to circleci-proxy.
• Workflow configurations that are stored in PostgreSQL will migrated to the Object Storage.

Known issues

• Telegraf needs to have its configuration overwritten and base64 encoded.
• When migrating from 3.x to 4.x, workflow configuration migrations will run as an init job and the workflows-conductor-event-consumer and the workflows-conductor-grpc-handler pods will not start until the migration completes. Customers with large PostgreSQL databases should plan accordingly. Migration time can be reduced by scaling the workflow-conductor pods.
• Webhook-service may not work with an externalized Vault.
• Vault may not refresh its client token after a month of uptime.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, users should be directed to use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.

To learn more about Server 4.x installation, migration, or operations please see our documentation.

What’s New

GitLab is now generally available for CircleCI. You are able to connect to GitLab and run pipelines from your GitLab repos. GitLab support also offers a new model for roles and permissions in CircleCI, independent of the VCS. This gives you full control over users and their access permissions.

We will continue to release GitLab-related features in Q3. These include project roles, restricted contexts, scheduled pipelines, and auto-cancel redundant workflows.

Each user can create up to three free organizations. A free organization can have up to 10 projects. If you need additional organizations or projects, consider upgrading to a paid plan, or contact our Support team.

What’s New

Today, CircleCI launched the CircleCI Public Roadmap, which showcases some of the product features and focus areas we are investing in. This product roadmap includes a set of features customers have expressed interest in and requested.

If you would like to provide feedback or suggestions on how we can improve the roadmap, please submit your request in the CircleCI Public Roadmap Ideas page.

What’s New

CircleCI now supports using Test Splitting on jobs that run on self-hosted runners. With this release, Runner jobs can take advantage of CircleCI’s parallelism and Test Splitting features in a manner identical to how jobs that run on CircleCI-hosted compute use the features.

For more detailed information on Test Splitting for Runner, please see the Canny and CircleCI Discuss posts.

What’s New

CircleCI has made it possible for you to access the billing portal without needing to be redirected away from the CircleCI website and on to Chargify’s portal. This eliminates any issues regarding the security of payment details.

The Billing Portal allows you to have complete access to your plan and payment settings including:

• making changes to your plan
• adding/removing credit packages
• adding coupons
• adding a new credit card
• viewing and downloading billing statements

With this update, you will now have access to all of this information in one single window that can be accessed directly through your CircleCI dashboard.

What’s New

The IP ranges feature is now generally available to all customers on a Performance or Scale plan. This feature is designed for customers who need to configure IP-based access to their restricted environments. As part of this feature, CircleCI provides a list of well-defined IP addresses associated with the CircleCI service. CircleCI jobs that have this feature enabled will have their traffic routed through one of these defined IP addresses.

As of January 28, 2022; the feature will incur 450 credits per GB of data usage for jobs that have IP ranges enabled.

Note: IP ranges is currently only available for Docker jobs, not including remote_docker.

For more detailed information about IP ranges, please refer to the IP ranges technical documentation.

For information on CircleCI plans, refer to the Pricing and Plans page.