CircleCI, makers of a popular cloud-based continuous integration platform, has warned developers that it was hit by a security “incident” and strongly recommends that any credentials stored in its system be changed. The company has also advised customers to check for “any unauthorized access” to their systems from December 21st 2022 through to the moment when the credentials are changed.
In addition, CircleCI has invalidated Project API tokens, used to obtain permissions to read and/or write data to CircleCI. Customers using these tokens will need to recreate them.
The post yesterday did not specify any details about the event but promised to provide “updates about this incident, and our response, as they become available.”
A customer discussion revealed several key points. A developer asked which secrets might be affected beyond those mentioned specifically in the CircleCI post, such as “SSH keys, Jira and Slack integration tokens, webhook secrets, etc?” The answer was that “all tokens and secrets” should be rotated – meaning deleted and recreated.
Another question was “is it safe to add new secrets in CircleCI?” The answer from an employee was that now “we are confident that there are no unauthorized actors active in our systems.”
Rotating all secrets may not be an easy task. “We have hundreds of repos and a variety of platform teams working in CircleCI, as a security team it’s difficult to ensure we have rotated everything,” said a developer. Another has now provided a script, published on GitHub, intended to list all credentials.
The hardest task though is the advice to “check the logs of any systems which had secrets stored in CircleCI.” Secret management for continuous integration and deployment is a tough problem, since it is necessary for such systems to have sufficient privileges to deploy software as well as perhaps other credentials for things like access to databases or other secure APIs. If these credentials were stolen, there may be unauthorized access to those systems which may in turn be exploited to access other systems.
Another developer asked: “Other than leaked credentials and secrets, is there any chance for attackers to have injected code or tampered with our builds?” There was no answer from CircleCI at the time of writing.
The implications of an incident such as this one will vary depending on the extent to which an organization is following best practice in terms of secret management and the principle of least privilege. A 2021 CircleCI post recommended using secret management tools, automated secret management, and to “cycle secrets on release.”
Just two months ago, Chief Technology Officer Rob Zuber also posted: “We’ve recently seen an increasing number of phishing attempts where unauthorized actors impersonate CircleCI to gain access to our users’ code repositories on GitHub. None of CircleCI’s systems have been compromised, and our customers’ data and information remain safe.”
Unfortunately it appears that at least between December 21st 2022 and January 4th 2023, this was not the case.