With CircleCI’s Evals orb 1.x.x, users were able to configure CircleCI to orchestrate evaluations of their LLM-enabled applications within their CI pipeline.

Evals orb 2.0.0 builds upon version 1.x.x, introducing Evaluation Testing. With Evals orb 2.0.0, users are able to automate the entire process of running evaluations, reviewing evaluation results and determining whether results meet application performance criteria. This removes the need to manually trigger evaluations and manually review evaluation results. Automating evaluations with CircleCI enables developers to automatically determine whether or not to merge proposed changes, or deploy their AI application.

To use Evaluation Testing, users configure test cases to determine if the results of an evaluation meet specified conditions. Each test case represents a scenario with an assertion to check if a specified condition is true or false. If any test case fails, the evaluation test fails. You can configure test case assertions such as Thresholds (Check whether 1 or more results fall within a range), and Equality (Check whether results provide an expected answer)

Evaluation Testing results are presented in the CircleCI web UI in two locations:

1. In the step details
2. In the tests tab

Runner Release 3.0.25

Both Machine and Container Runner:

• Sign docker manifests to be verified with the CircleCI Public Key
• Security fixes
• Increase time between deleting existing task-agents to prevent unexpected removals during task execution

Machine Runner:

• Set KillMode=process for linux based machine runners

We have added release markers and release jobs to CircleCI Releases!

Release markers provide a lightweight way for users to generate a changelog and easily track their releases without requiring a full CircleCI Releases setup (no agent install needed!). Release markers can be configured for releases to any deployment target (ie. ECS, EC2, VMs, k8s, etc). See our documentation for information on setting up release markers.

Release jobs replace kubernetes manifest annotations as the way to configure deployment triggers. Release jobs also allow users to monitor their kubernetes release process as part of the overall deployment workflow in CircleCI, making it easier to find releases for a given pipeline. Users can also use release jobs to orchestrate releases across multiple environments (for example, deploy to production only after a successful release in a staging environment). See our documentation for information on setting up release jobs.

We’ve added additional meta data using the “tag” key to audit logs to help customers identify how workflow reruns were triggered.

Workflow tags include:

• setup
• rerun-single-job
• rerun-workflow
• rerun-workflow-from-beginning
• rerun-workflow-from-failed
• rerun-workflow-from-job-ids-sparse-tree
• rerun-single-job-with-ssh

Example audit log entries

Workflow Started:

workflow.start
{"job":{},
 "workflow":{"id":"247...","name":"test-and-build","tag":null}}

workflow.job.context.request
{"job":{"context_uuids":[],"contexts":[],"id":"408...","job_name":"test","job_status":"pending"},
 "workflow":{"id":"247...","name":"test-and-build","tag":null}}

workflow.job.start
{"job":{"contexts":[],"id":"408...","job_name":"test","job_status":"running"},
 "workflow":{"id":"247...","name":"test-and-build","tag":null}}

workflow.job.finish
{"job":{"contexts":[],"id":"408...","job_name":"test","job_status":"failed"},
 "workflow":{"id":"247...","name":"test-and-build","tag":null}}

Retry the original workflow with rerun job with SSH enabled:

workflow.retry
{"job":{},
 "workflow":{"id":"247...","name":"test-and-build","tag":null}}

SSH enabled rerun started:

workflow.start
{"job":{},
"workflow":{"id":"811...","name":"test-and-build","tag":"rerun-single-job-with-ssh"}}

workflow.job.context.request
{"job":{"context_uuids":[],"contexts":[],"id":"c6a...","job_name":"test","job_status":"pending"},
 "workflow":{"id":"811...","name":"test-and-build","tag":"rerun-single-job-with-ssh"}}

workflow.job.start
{"job":{"contexts":[],"id":"c6a...","job_name":"test","job_status":"running"},
 "workflow":{"id":"811...","name":"test-and-build","tag":"rerun-single-job-with-ssh"}}

workflow.job.finish
{"job":{"contexts":[],"id":"c6a...","job_name":"test","job_status":"canceled"},
 "workflow":{"id":"811...","name":"test-and-build","tag":"rerun-single-job-with-ssh"}}

The pipelines page can load slowly for some users. The number of pipelines and number of projects can contribute to that slowness. A recent change to one of the underlying data services has been made resulting in a 30%-40% improvement in performance.

We continue to work on improving the pipelines page experience. Our current focus is on load time. We do have plans to introduce new filter and search capabilities in the future.

We have added an API and UI component to approval jobs that allows you to “cancel” the approval job. Previously it was not possible to cancel approval jobs. This limitation forced you to either approve the job or let it expire, which takes 90 days. This led to a suboptimal experience with jobs left in a non-terminal state. This could impact performance metrics, as well pipelines views. It also led to non-intuitive actions being taken like cancelling the workflow which cancels all remaining jobs.

We are following up this feature with more flexible require statements. So you can have a job depend on an approval while another depends on cancellation. This is great for cleanup jobs, notifications, or other tasks that may only need to happen or not happen upon cancellation.

Previously, when processing a configuration file, CircleCI would process all workflows defined in the workflow. Upon completion of processing, errors such as missing or undefined parameters would be included in feedback.

CircleCI will no longer process workflows defined in a configuration file that aren’t going to be run. This means errors that might exist within that workflow will also not be caught and reported back to the user.

During development or troubleshooting, users should ensure a workflow has run, and it’s free of any errors. This can be done by making sure any conditionals like when: <complex condition> are replaced with when: true.

As previously reported https://circleci.com/changelog/launch-agent-1-x-variants-eol/.

Support for all 1.x variants of runner launch agent has ended. It has been replaced with Machine Runner 3.0. Any remaining access will be removed September 17th, 2024. For those that haven’t migrated, your builds on runner will fail.

You can learn more about machine runner and migration here.

For CircleCI organizations that integrate with GitLab or GitHub App:

• The user count on the People tile reflects the number of users in the organization instead of active users in the billing period.

• Organization admins have the ability to invite additional teammates directly from the Organization Homepage by clicking the “Invite your teammates” button.

The CircleCI Automation Actor will be used to attribute automated workflow and job cancelations on September 17th, 2024. This change will impact two functionalities:

• Auto-cancel redundant workflows
• Jobs canceled via the v1 Cancel Job API using a project token

This change is intended to better indicate that the action was performed by a machine.

Once the change is completed, the canceled_by field of the getWorkflowById and the listWorkflowJobs API response will return the CircleCI Automation Actor ID. You only need to take action if you rely on this field to have a static value.

Users who had more than ~200 repositories in their GitHub organization & integrated with CircleCI’s GitHub App were hitting a timeout error when setting up a new project that required them to select one of their repositories.

To resolve the timeout, we reduced the number of repositories displayed in the CircleCI UI to be the 100 most recently used repositories.

We have just released a change to the project creation flow in the CircleCI UI that enables searching through all GitHub repositories without any timeout errors, not just the 100 most recently used repositories.

We will be removing the vcs_branch field from the payload column in following audit events:

• workflow.start
• workflow.error
• workflow.job.finish
• workflow.job.context.request
• workflow.schedule.start

The field has been moved to a new event associated with each pipeline named trigger_event.create.

Audit logs are exported in CSV format. Currently workflow.start event payload column looks like this:

{“job”:{},”workflow”:{“id”:”00000000-0000-0000-0000-000000000000”, “name”:”test-and-build”,”vcs_branch”:”my-feature-branch”}}

After September 21, 2024, any future event will not include the vcs_branch field, all existing events will remain unmodified:

{“job”:{},”workflow”:{“id”:”00000000-0000-0000-0000-000000000000”, “name”:”test-and-build”}}

The VCS data has been moved to a new trigger_event.create event payload column that is recorded for each created pipeline:

{“pipeline_id”:”00000000-0000-0000-0000-000000000000”, “config_source”:”vcs”, “trigger_source”:”webhook”, “branch”:”my-feature-branch”, “tag”:”my-tag”}

If the vcs_branch field in the payload column is used, customers will need to update their usage to point to the new trigger_event.create, otherwise no action is required.

Previously: The pipeline.trigger_parameters.webhook.body pipeline value was only populated when a custom webhook triggered the pipeline. If a config used pipeline.trigger_parameters.webhook.body and the pipeline was triggered with a GitHub App event, an errored pipeline would occur. This made it difficult to use one configuration file with a custom webhook & GitHub App triggers.

New behavior: The pipeline.trigger_parameters.webhook.body pipeline value will be populated with an empty string if the pipeline is triggered from a GitHub App event. This simplifies using one configuration file with a custom webhook & GitHub App triggers.

On or shortly after August 30, 2024, all new users who sign up for CircleCI with an email/password account (the default sign-up method) will be required to verify their email immediately after CircleCI account creation.  

Users will not be able to proceed into the CircleCI application unless they verify their email. This will apply to users who are creating new orgs and users who are joining their teammates in an existing CircleCI organization.

CircleCI machine images have been updated with the latest software versions. Learn more about these changes:

• Ubuntu
• Windows
• Android

Users who had more than ~200 repositories in their GitLab.com organization were hitting a timeout error when setting up a new project that required them to select one of their repositories.

To resolve the timeout, we reduced the number of repositories displayed in the CircleCI UI to be the 100 most recently used repositories.

We have just released a change to the project creation flow in the CircleCI UI that enables searching through all GitLab repositories without any timeout errors, not just the 100 most recently used ones.

Before Upgrading

See the CircleCI server 4.6 release notes and upgrade guide for this release.

NOTE: Vault is being deprecated and will no longer be supported in server 5.0. Refer to our script for steps to migrate to Tink.

What’s New in Release 4.6.0

The v4.6 release introduces various UI refreshments and improvements.

NEW FEATURES

• Several enhancements to the CircleCI web app, including:
  • A new “User Homepage”.
  • Updated styling for the left-hand navigation bar.
  • Top navigation bar that allows easy access to user-level actions.
• Pipelines are now labeled whenever they are rerun from start or fail, or with SSH.
• Project-based context restrictions. For more information see the docs.

CHANGES

• Old, expired workflows are now automatically canceled after 90 days.
• The pipeline.in_setup pipeline parameter has been removed.
• Kong, which is used for server API management, has been upgraded from 2.x to 3.x.

BUG FIXES

• Fixed a bug that prevented jobs from being rerun with SSH.
• Resolved an issue that prevented project loading on the Insights dashboard.
• Addressed regressions from server 4.2:
  • Introduced a configuration option at the organization level to set the default resource class for a Docker job. In server 4.3, this was previously hardcoded to large.
  • Added support for absolute paths in defining a GCP network under the machine provisioner configuration. This feature is particularly useful for VM jobs using shared networks, enabling explicit network selection. This corrects a regression from the previous VM service architecture.

NEW SERVICES

New services introduced with this release:

• authentication-service - provides an interface for user and identity management in the CircleCI system

DATABASE MIGRATIONS

The following databases will run migrations when upgrading to this version:

• builds_service
• conductor_production
• permissions

KNOWN ISSUES

• SSH reruns in air-gap will time out, leaving the job in an error state
• Vault may not refresh its client token after a month of uptime. Migrate to Tink to resolve this issue.
• Retry with SSH for jobs using the machine executor advertises a private IP address. For this reason, retry with SSH for jobs using the machine executor works as standard for public installations, but for private installs you would need to ensure that you can access the private IP advertised. For example, by using a VPN into your VPC.
• CircleCI 1.0 builds are not supported. If an attempt is made to run a 1.0 build, no feedback will be available in the application to indicate the cause of the issue. If a build is run on your installation and does not show up in the CircleCI application, use the CircleCI CLI to validate the project configuration and get details of the possible cause of the issue.
• Open ID Connect Tokens are not generated and not available.

To learn more about Server 4.6 installation, migration, or operations please review our documentation.

An experimental feature to filter notifications by branch “notify” is now deprecated. It will be removed in 30 days September 8th, 2024. Once this change is completed customers will continue to receive notifications but they will no longer be filtered by branch.

No action is required The non-functional config code will not cause compilation errors. We do recommend removing the code from your config.

Example

experimental:
  notify:
    branches:
      only:
        - deployed
        - master

Three new fields will be introduced to the Usage API on August 23, 2024 to help you better understand how your builds are running. These fields are:

• job_run_queued_at: the timestamp (UTC) of when the job started to queue (if applicable)
• job_run_started_at: the timestamp (UTC) of when the job started to run
• job_run_stopped_at: the timestamp (UTC) of when the job stopped (only logged for success and failed job build statuses)

These three new fields will be introduced into the CSV output in the order above immediately following the job_run_date and prior to the job_build_status fields.

NOTE: If you have an automated ingestion process that relies on the format of the CSV remaining consistent, please update your ingestion process to accommodate these three new fields prior to August 23, 2024.

What’s changed?

• Authenticated users are now redirected from /server-login to the app subdomain’s dashboard.
• Users who require admin approval before logging in will now see an error message on the login screen.
• Improved error handling when retrieving pipeline values from GCS
• Default resource class can now be set via Repl.
• Project-based context restrictions now populate the project list.
• ~Improved k8s runner start up times. Task agent will now be initialized via init image rather than waiting to copy over the agent after the task pod has started.~

Update: Aug 12, 2024. We discovered that the init container changes were not included in this release and have updated the change log.

Fixes

• Fix for rerun with SSH docker and machine jobs
• Fixed broken insights page
• Fix to allow absolute paths or GCP networks
• Fix to address changes in the GitHub API that broke webhook setup for projects

Correction

• Fix for rerun with SSH docker with machine jobs is not resolved

In order to prevent timeouts for customers with a large number of repositories, dropdown menus for repository search for GitLab and GitHub App now limit results to 100 repositories.

For Gitlab projects, entries are ordered by most recently updated. For Github App projects, entries are order alphabetically.

This is a temporary solution. Search functionality will be introduced soon to allow selection of repositories that don’t appear among the first 100 results.

In the meantime, GitHub App users running into issues may configure the CircleCI GitHub App to grant access to a selection of individual repositories (up to 100) rather than selecting the option “All repositories”. This will ensure that all selected repositories will appear in the CircleCI dropdown menu.

For any issues, please contact benedetta@circleci.com.