On October 29th, we received a notification email from our database provider telling us that they had been compromised. We later learned that our database was among those accessed by the attackers.
As soon as we learned this, we immediately moved to protect user data as much as possible by shutting down our site and our builds, revoking any SSH keys and API tokens to which we had access, and working with upstream providers to revoke all compromised security tokens.
We have no evidence that these keys and tokens were exploited, and shut down the service as a preventative measure to protect your data. Below we describe how you can check if your systems were exploited as a result.
We are incredibly sorry that this happened. Your security is paramount to the entire team at CircleCI, and once we were aware of the issue we took immediate steps to protect you and maintain your trust.
We will continue to work with you to provide as much detail as possible about the incident, our response, the actions we will be taking as a result, and how to protect yourselves going forward.
Please feel free to contact us for specific information about your code, or any more detail about how you can protect yourself from this incident.
On Monday, October 28th, at 3:46pm Pacific, we noticed an IAM key had been deleted. We immediately searched for any evidence that our systems had been compromised, and to discover the source of the key deletion. As a precaution, we began cycling our keys and security credentials.
On Tuesday, October 29th, at 2:07pm, MongoHQ announced they had been compromised. In the announcement, we were told that there was no evidence that our DB was accessed by an unauthorized user, as we had not been alerted to it.
At 5:03pm, we learned from MongoHQ that our database was accessed by one of the IPs responsible for the intrusion. The access occurred late at night UTC on October 27th, two days previously.
We immediately initiated a full response to assess the exposure and determine the best way to protect our users.
To contain any potential risk we determined that the best course of action was to shutdown the CircleCI website and stop all builds. We also determined that it would be best to revoke all API tokens and SSH keys that we had access to, and work with upstream vendors to similarly protect users from possible exposure.
At 7:13pm, we began moving forward with our plan. We shut down the CircleCI website and redirected it to our status page, normally located at https://circleci.statuspage.io/. We then stopped all the builds running at that time and disabled all scheduled builds and GitHub hooks.
At 7:30pm we had completed these actions and posted an update to our status page.
Our goals were to protect our customers, to communicate what had happened with them, and only then to recover the CircleCI service after we were certain of the safety of customer code.
We then began working on all potentially affected GitHub repositories. At 7:20pm we contacted GitHub security to request that the relevant OAuth tokens and SSH keys be revoked.
By 9:35pm GitHub had revoked all CircleCI OAuth tokens. We continued to work with GitHub, providing lists of affected customers and SSH keys. By 11:28pm all known SSH deploy keys were revoked, and all remaining SSH user keys were revoked by 11:43pm.
At 9:05pm we sent Heroku a list of all potentially affected Heroku API tokens. By 9:32pm, Heroku had revoked all their tokens.
At 9:37pm we contacted Amazon with a list of potentially affected IAM keys and requested that they initiate their process to revoke them. We additionally began notifying other potentially affected providers so they could take similar actions.
At 11:26pm we sent Heroku a list of all known SSH keys. We learned at 1:25pm today that those keys have all been revoked.
At 11:44pm we completed the process of deleting all our caches and cycling all our keys.
At 2:49am, we sent additional notifications to all CircleCI users who may have stored SSH keys in CircleCI’s database, along with the relevant keys, and provided instructions on the actions they should take.
What you should do
In order to protect your data and users, we strongly urge you to secure each of these systems:
systems accessible by SSH keys uploaded to CircleCI
systems accessible using API tokens stored in CircleCI env vars
systems accessible using an API token or SSH key stored in a GitHub repo accessible from your CircleCI account.
In addition, you should validate that your applications and code are unaltered:
systems accessible for deployment from CircleCI may have been compromised
your git repos may have been written to – you should check that all sha1s are as expected
you should validate your Heroku account’s environment variables and database credentials
To determine whether your systems have been accessed, contact GitHub, Heroku, your hosting provider, or consult your own logs. We have provided GitHub and Heroku with recent IP addresses of CircleCI servers, and the malicious servers that MongoHQ believes compromised their databases.
We take security seriously. We appreciate your trust and have done everything possible to protect your data. Along with the entire CircleCI team, I am sorry that this unfortunate incident has affected you. We will continue to work hard to keep your data and systems secure. If you have any questions, please contact me at firstname.lastname@example.org.