When security incidents happen, it’s crucial for software providers and users alike to take swift and effective action. In response to our recent security incident, we witnessed firsthand how an open and collaborative effort between our customers, technology partners, and engineering teams helped to contain the threat and mitigate risk of unauthorized access to customer systems.
In this post, we’d like to highlight some of the new features and tools we produced in response to the January 4, 2023 security incident. These include:
- An interactive CLI tool for identifying all secrets stored in CircleCI for you organization’s projects
- Changes to the CircleCI UI and API for easier secrets auditing
- Universal access to self-serve audit logs to review changes to your CircleCI environment over a 30-day window.
Together, these updates make it easier for customers to rotate secrets via our UI or API and to verify that all affected pipelines and credentials have been secured. In many cases, these changes were inspired by the questions, requests, and proactive responses of our community members as they worked to reset credentials and monitor activity across their projects and connected services. We are thankful for your contributions and continued support in responding to this incident.
Identify secrets with the CircleCI-Env-Inspector
The CircleCI-Env-Inspector is a command line interface (CLI) tool that you can use to generate a JSON report of all secrets you have stored on CircleCI. This tool is particularly useful for organizations with a large number of secrets in project environment variable and org-wide contexts as it provides a comprehensive list of items that require manual rotation. Note that the tool will only return names and other identifying information, but will not return the secret values, meaning there is no risk of accidental exposure of sensitive information.
To use the Env-Inspector tool, visit and clone the project repository, then generate a CircleCI API token with access to all projects you want to inspect. To list secrets used in all projects across your entire organization, you will need to generate the API token from an org admin account.
Once you have your API token, navigate to the root of the Env-Inspector repository and run the command run.sh. Enter your API token at the prompt, and the tool will return a list of all secrets stored in project environment variables and contexts. Below is an example output:
{
"contexts": [
{
"name": "CONTEXT_NAME",
"url": "https://app.circleci.com/settings/organization/<VCS>/<ORG>/contexts/<CONTEXT-ID>",
"id": "xxx",
"variables": [
{
"variable": "GITHUB_TOKEN",
"context_id": "xxx",
"created_at": "yyy"
}
]
}
],
"projects": [
{
"name": "ORG/REPO",
"url": "https://app.circleci.com/settings/project/<VCS>/<ORG>/<REPO>/environment-variables",
"variables": [{ "name": "VAR", "value": "xxxx" }],
"project_keys": [
{
"type": "deploy-key",
"preferred": true,
"created_at": "xxx",
"public_key": "yyy",
"fingerprint": "zzz"
}
]
}
]
}
The output includes several pieces of information you can use to identify secrets stored across your projects, including the name of the context or project, URL, ID number, and the date and time at which the credential was created and last updated.
The CircleCI Env-Inspector is an open source tool, and we welcome community contributions in the form of issues or pull requests on the project repository.
Update - 13 March 2023: We have updated the Env-Inspector tool to further aid in discovery of secrets that are stored in CircleCI but may not be visible in the UI. Examples include variables for renamed projects or projects deleted from GitHub but not from CircleCI.
Confirm secrets have been rotated with updated_at timestamps
To give customers peace of mind that their long lived secrets have been changed, we introduced updated_at timestamps to the Context APIs and environment variable and context management pages in the web app. This information allows you to verify that secrets have been successfully rotated and to identify any secrets that need to be updated.
In the API, updated_at data is provided in the response for both a GET request to list context environment variables and a PUT request to add or update existing context variables. Below is a sample response from an update request to the contexts API:
{
"variable" : "a_variable",
"updated_at" : "2023-01-09T20:32:18.568Z",
"context_id" : "b5a5561a-7159-48f1-854f-1e53a8f315e0",
"created_at" : "2023-01-09T20:26:57.020Z"
}
The response includes the name of the variable, the date and time at which it was updated, the ID number of the context in which the variable is stored, and the original date and time at which the variable was updated.
For users who prefer to update secrets via the web app, we also updated our contexts UI to provide information on when variables were created and last updated.
To view context environment variables for your organization, visit Organization Settings > Contexts.
Compare SSH checkout keys with SHA256 fingerprints
As part of our remediation guidance to customers, we advised rotating all SSH keys both in CircleCI and in your target environments. To make it easier for you to compare checkout keys across systems, we added the ability to return SHA-256 public key fingerprints from our v1.1 API. This is in addition to the existing ability to return MD5 fingerprints.
Below is a sample API call that retrieves a SHA-256 identifier:
curl -H “Circle-Token: " https://circleci.com/api/v1.1/project/:vcs-type/:username/:project/checkout-key?digest=sha256
The ?digest=sha256 query parameter specifies that the response will return the SHA-256 hash of the checkout key.
For more information on rotating SSH keys, visit our support article.
Review activity in your CircleCI environment with self-serve audit logs
To assist customers in auditing and analyzing activity under their organization, CircleCI records important system events in audit logs. Customers can access these logs in two ways:
- Through a request to customer support
- Directly from a self-service portal in the web app
Traditionally, self-serve audit logs were available only to customers on our scale, performance, and server plans. To assist all customers in securing their systems, we have temporarily extended self-serve capabilities to include customers on our free plan.
Users can request logs of system activity for any 30-day period in the previous 365 days by visiting Organization Settings > Security and selecting a date range. Customers on our free plan can request one audit log per day, while customers on paid plans can request up to three logs per day.
CircleCI audit logs record important system activities such as the creation or deletion of contexts, project environment variables, SSH keys, or API tokens; when workflow jobs are started, completed, or approved; and changes to scheduled pipelines. The logs include data on when the action was taken, which user initiated it, which entity (org, project, account, or build) was affected by the action, and whether the action was successful.
Please note that while reviewing your CircleCI audit logs is an important step in ensuring no unauthorized activity occurred in your CI/CD environment, we strongly recommend that you also review audit logs for all connected services used in your build processes between the dates of December 21, 2022 and January 4, 2023.
Conclusion
The new features and updates highlighted in this post were produced by our engineering teams to immediately assist customers in responding to the January 4 security incident. Regularly rotating static credentials, however, is a security best practice, and we encourage you to continue using these features as part of your normal security routines. Additional steps you can take to secure your pipelines include:
- Using short-lived credentials like OpenID Connect authentication tokens in place of static credentials wherever possible
- Using the CircleCI CLI or API to automatically rotate any static credentials stored in contexts
- Restricting inbound access to your build environments with IP ranges
- Running sensitive workloads on private infrastructure using self-hosted runners
We are grateful to our customers for your ongoing collaboration and vigilance in keeping your pipelines secure, and our teams are here to support you. If you have any questions regarding the incident, our ongoing response, or our recommendations to customers, please visit our complete incident report or reach out to customer support.
