CircleCI security alert: Rotate any secrets stored in CircleCI (Updated Jan 13)

Security update 01/13/2023 - 21:25 UTC

Our full incident report is now available. Read the Report

Security update 01/12/2023 - 00:30 UTC

We have partnered with AWS to help notify all CircleCI customers whose AWS tokens may have been impacted as part of this security incident. Today, AWS began alerting customers via email with lists of potentially impacted tokens. The subject line for this email is [Action Required] CircleCI Security Alert to Rotate Access Keys.

Our goal in working with AWS on this additional level of communication is to help customers more easily identify and revoke or rotate any potentially affected keys. For assistance, please see AWS documentation on rotating access keys or reach out to Amazon support.

Additional questions you may have:

• If I received the email, does this mean someone gained unauthorized access to the AWS account listed? At this time, there is no indication that your AWS account was accessed, only that there is a possibility the token stored in CircleCI was leaked, and therefore should be deleted from AWS and rotated.

• What’s new here since CircleCI disclosed on January 4? Has something else happened? This is an additional alert as part of the original disclosure CircleCI made on January 4, 2023. No new information or additional developments have come to light. This note is in service of aiding customers in identifying and rotating AWS tokens on AWS.

Security update 01/10/2023 - 21:10 UTC

This is a short update to communicate the status of our incident report. We expect to provide an incident report to our customers on Tuesday, January 17 (PST).

We have confidence in the security of the CircleCI platform, and customers can continue to build. Our support engineering, customer success, and security teams continue to stand by to assist you with any questions or concerns. We are also continuing to connect with our customers and community on our forums here. Thank you.

Security update 01/07/2023 - 07:30 UTC

Yesterday, we let customers know that we were in the process of rotating GitHub OAuth tokens on behalf of customers. That process is now complete, and all GitHub OAuth tokens have been rotated.

Customers who wish to rotate their own OAuth tokens may still do so following the directions outlined below.

At this point, we are not expecting to have additional substantive updates to share until we have completed our ongoing investigation with our third-party forensic team. We have confidence in the security of the CircleCI platform, and customers can continue to build.

We want to continue to express our appreciation and consideration for our customers. We know that secret and variable rotation can be time-consuming, painstaking work. Thank you for your support and care in keeping all systems safe. Our team at CircleCI has been working around the clock to perform as many mitigating actions as possible in support of our customers. Our support engineering, customer success, and security teams are available to help. We are also continuing to connect with our customers and community on our forums here. Thank you.

Security update 01/06/2023 - 23:00 UTC

This is a short update to provide the status of our GitHub OAuth token rotation. As of 23:00 UTC on January 6, 2023, we are 99% complete in token rotation. We expect to be complete in the coming hours and will provide an additional update upon completion.

As a point of clarification, for customers working on rotating secrets and keys, you should rotate keys at the source (the system to which they provide access) and then store the new secrets on CircleCI. Simply removing them from CircleCI is not enough.

Thank you for your support and care in mitigating this issue. Our support engineering, customer success, and security teams continue to stand by to assist you with any questions or concerns. We are also continuing to connect with our customers and community on our forums here. Thank you.

Security update 01/06/2023 - 17:52 UTC

Our team is working to take every action available to assist customers in the mitigation of this incident.

Since our last update, our team has addressed the following areas on behalf of customers:

• Personal and Project API Tokens: We have removed all Personal and Project API Tokens created before 00:00 UTC on January 5, 2023.
• Bitbucket OAuth: As of 10:00 UTC on January 6, 2023 our partners at Atlassian expired all OAuth tokens for Bitbucket users. Bitbucket tokens will refresh for users upon login, and no additional action is needed here. Bitbucket users will still need to replace SSH tokens.
• GitHub OAuth: As of 07:30 UTC on January 7, all GitHub OAuth tokens have been rotated on behalf of CircleCI customers. Customers who wish to rotate their own GitHub OAuth tokens may follow the directions below.

Note: This security incident is not relevant for customers using CircleCI server.

Updated instructions as of 2:00 UTC on January 10

1. OAuth tokens:
   1. For GitHub: As of 07:30 UTC on January 7, all GitHub OAuth tokens have been rotated on behalf of CircleCI customers. Customers who wish to do so may rotate their own OAuth tokens by logging out of the CircleCI application, going to https://github.com/settings/applications, selecting “Authorized OAuth Apps”, and then revoking the CircleCI entry. Once that’s done, log back into CircleCI to trigger reauthorization.
   2. For Bitbucket: As of 10:00 UTC on January 6, 2023 our partners at Atlassian expired all OAuth tokens for Bitbucket users. Bitbucket tokens will refresh for users upon login, and no additional action is needed here. Bitbucket users will still need to replace SSH tokens.
   3. For GitLab: GitLab users do not need to reauthorize their application access. As a precautionary measure, we would still recommend GitLab users rotate their environment variables, Personal and Project API tokens, and all SSH keys.
2. Project API tokens: To rotate them, go to Project Settings > API Permissions > Add API Token. Update: CircleCI has revoked all tokens created before 00:00 UTC on January 5, 2023.
3. Project environment variables: Go to Project Settings > Environment Variables and then create an environment variable with the same name to replace the existing value.
4. Context variables: Go to Organization Settings > Contexts and do the same thing as for project environment variables for each context.
5. Note: As of 22:50 UTC on January 9, we have updated the Contexts API to include the last “updated_at” date and time stamp. This gives the necessary information to determine if secret rotation was successfully completed. We will be rolling out additional changes to ensure the updated_at date is included in the UI, in addition to the API. You can read more in API documentation contexts and environment variables.
6. User API tokens: Go to User Settings > Personal API Tokens and then delete and recreate any tokens you might be using. Update: CircleCI has revoked all tokens created before 00:00 UTC on January 5, 2023.
7. Project SSH keys:
   1. Go to Project Settings > SSH Keys.
   2. Delete the Deploy Key and add it again.
   3. If you were using any additional keys, then those need to be deleted and recreated.
   4. Note: SSH keys will also need to be rotated from the target environment.
8. Runner Tokens: using the CircleCI CLI, run the following commands:
   1. circleci runner token list <resource-class name>
   2. circleci runner token delete "<token identifier>"
   3. circleci runner token create <resource-class-name> "<nickname>"
   4. Following these commands, you will need to add the created token to your launch-agent-config.yml and restart your runner service

Note: there is also a tool for discovering all your secrets on CircleCI that can be used to find an actionable list of items for rotation.

Update - 9 January 2023: We have added the functionality to return SHA256 signature for checkout keys using our get-checkout-key API V1.1 endpoint.

Please see below for a example API Call :

• curl -H “Circle-Token: " https://circleci.com/api/v1.1/project/:vcs-type/:username/:project/checkout-key?digest=sha256
• Please note the sha256 query parameter here

Update - 13 March 2023: We have updated the tool to further aid in discovery of secrets that are stored in CircleCI but may not be visible in the UI, e.g. variables for renamed projects or projects deleted from GitHub but not from CircleCI.

We appreciate your adaptability and commitment to keeping systems safe and secure. We will continue to provide updates and information here as more details become available. We are also continuing to connect with our customers and community on our forums here. Thank you.

Security update 01/05/2023

We wanted to update customers about the security incident we disclosed yesterday, and provide additional clarity around commonly asked questions from our customers. We understand that starting the New Year with this disruption to your work is not ideal. We appreciate your patience and understanding as we work together to keep all systems secure.

CircleCI customers can build

The number one question we’ve received from customers is, “Can I build?” The answer is yes.

We are confident that we have eliminated the risk that led to this incident. We’ve taken the following actions to ensure the integrity of our platform:

• We have rotated all production machines and cycled all access keys.
• We have completed an audit of all system access.
• We are actively working with third-party investigators and our partners to validate the steps and actions of our investigation.

What you should continue to do now

We also want to provide more details on our recommended actions for all customers.

Please rotate any and all secrets stored in CircleCI. There are multiple ways to do this, and we encourage you and your teams to use your preferred methods. Here is an approach you may follow:

1. OAuth tokens:
   1. For GitHub, this means going to https://github.com/settings/applications, selecting “Authorized OAuth Apps”, then revoking the CircleCI entry. Once that’s done, log out and back into CircleCI to trigger reauthorization.
   2. For BitBucket: https://bitbucket.org/account/settings/app-authorizations/.
   3. For GitLab: GitLab users do not need to reauthorize their application access. As a precautionary measure, we would still recommend GitLab users rotate their environment variables, Personal and Project API tokens, and all SSH keys.
2. Project API tokens: To rotate them, go to Project Settings > API Permissions > Add API Token.
3. Project environment variables: Go to Project Settings > Environment Variables and then create an environment variable with the same name to replace the existing value.
4. Context variables: Go to Organization Settings > Contexts and do the same thing as for project environment variables for each context.
5. User API tokens: Go to User Settings > Personal API Tokens and then delete and recreate any tokens you might be using.
6. Project SSH keys: Go to Project Settings > SSH Keys. Delete the Deploy Key and add it again. If you were using any additional keys, then those need to be deleted and recreated.
7. Runner Tokens: using the CircleCI CLI, run the following commands:
   1. circleci runner token list <resource-class name>
   2. circleci runner token delete <token identifier>
   3. circleci runner token create <resource-class-name> "<nickname>"
   4. Following these commands, you will need to add the created token to your launch-agent-config.yml and restart your runner service

Note: We recommend this for all projects (go to “Project Settings”), orgs (go to “Organization Settings”), and users (go to “User Settings”).

In addition to these instructions, today, we created a tool for discovering all your secrets on CircleCI. This should assist you in creating an actionable list of items for rotation.

Additional security recommendations

While customers are in the process of rotating keys, secrets, and variables, it may be helpful to add additional layers of protection to your CI/CD pipeline configuration.

Here are a few things all customers can utilize to increase pipeline security:

• Use OIDC tokens wherever possible to avoid storing long-lived credentials in CircleCI.
• Take advantage of IP ranges to limit inbound connections to your systems to known IP addresses.
• Use Contexts to enable the sharing of environment variables across projects, which can then be rotated automatically via API.
• For privileged access and additional controls, you may choose to use runners, which allow you to connect the CircleCI platform to your own compute and environments, including IP restrictions and IAM management.

Addressing customer concerns

We also wanted to address a few customer concerns about how this incident is being handled. As a company deeply invested in iteration and improvement, feedback from our customers on incident management is always welcome and appreciated.

Here are a few questions customers have shared:

• How can I access my audit logs for CircleCI? We have expanded access to self-serve audit logs to all customers, including free customers. Customers can access self-serve audit logs via our UI. Customers can query up to 30 days of data and have 30 days to download the resulting logs. While we understand the requests for access to CircleCI audit logs, our recommendation to all customers is to focus your audits and investigations on the logs of any systems which had secrets stored in CircleCI.

• Why did you send this out so late on Wednesday? We understand that many of our North American customers experienced late nights and on-call rotations once our guidance to rotate secrets was released at 6:30 pm PT / 9:30pm EST on Wednesday, January 4. We erred on the side of getting information out as fast as possible to minimize any potential exposure time. We also know that as a global company with customers in almost every country, there is no good time to disclose a security incident except “as fast as possible.”

• Is this incident related to the December 21 reliability update you posted on your blog? No. As part of our customer communications, we provide regular updates on reliability. Our advice to review logs between December 21 and January 4 as part of this incident and the timing of our reliability update on December 21 is a simple coincidence.

Conclusion

We want to thank our customers and community for your support as we all work to keep our systems safe. We know that security incidents are stressful and burdensome for all involved, and for that, we apologize. We will continue to provide updates and information here as more details become available. We are also continuing to connect with our customers and community on our forums here.

Security alert 01/04/2023

We wanted to make you aware that we are currently investigating a security incident, and that our investigation is ongoing. We will provide you updates about this incident, and our response, as they become available. At this point, we are confident that there are no unauthorized actors active in our systems; however, out of an abundance of caution, we want to ensure that all customers take certain preventative measures to protect your data as well.

Action request

Out of an abundance of caution, we strongly recommend that all customers take the following actions:

• Immediately rotate any and all secrets stored in CircleCI. These may be stored in project environment variables or in contexts.
• We also recommend customers review internal logs for their systems for any unauthorized access starting from December 21, 2022 through today, January 4, 2023, or upon completion of your secrets rotation.

Additionally, if your project uses Project API tokens, we have invalidated those and you will need to replace them. You can find more information on how to do that in our documentation here.

We apologize for any disruption to your work. We take the security of our systems and our customers’ systems extremely seriously. While we are actively investigating this incident, we are committed to sharing more details with customers in the coming days.

Thank you for your urgent attention to rotating your secrets.