DevOps player CircleCI has published details of the security incident that forced it to warn cloud customers to revoke all secrets, tokens and credentials stored in its systems, as well as checking for any unauthorized access via those credentials.
A troubling aspect of the incident is that some CircleCI customers, though the company said “fewer than 5,” have reported “unauthorized access to third-party systems as a result.” An updated post listed IP addresses used by the attacker which may help log analysis, as well as VPNs used and a short list of “malicious files to search for and remove.”
CircleCI also advises customers to look for unexpected commands in GitHub audit logs such as “repo.download_zip,” suggesting that entire repositories of source code may have been stolen.
The attack began, CircleCI said, on December 16th 2022 when malware on an employee’s laptop was used to “steal a valid, 2FA-backed SSO [Single Sign On] session.” CircleCI has not explained how the laptop was compromised, but said that it “was not detected by our antivirus software.” The laptop belonged to an engineer who had “privileges to generate production access tokens” and by stealing the session cookie the criminals were able to “impersonate the targeted employee in a remote location and then escalate access to a subset of our production systems.”
CircleCI believes the cyber intruders “engaged in reconnaissance activity on December 19th” and then “was able to access and exfiltrate data from a subset of databases and stores, including customer environment variables, tokens and keys.” Although the data was encrypted at rest, it likely did not help because “the third party extracted encryption keys from a running process.”
The company became aware of the issue 10 days later, on December 29th, when a customer alerted CircleCI to suspicious access to GitHub via an OAuth token. Action taken included shutting down access for the affected employee and then for almost all employees, on January 4th; rotating “all potentially exposed production hosts,” revoking all project API tokens and personal API tokens, rotating Bitbucket tokens and GitHub OAuth tokens on behalf of customers, and asking AWS to notify customers of “potentially affected AWS tokens.”
Security was stepped up following the incident, blocking the “specific behaviours exhibited by the malware,” restricting access to production environments, and adding additional authentication steps, the company said.
Only cloud-hosted CircleCI accounts were affected. Developers using CircleCI hosted on-premises were not impacted.
CircleCI’s report shows that this attack was both stealthy and effective. It does not provide any clues to the motives of the attackers and since the criminals were able both to steal source code and to access third-party systems they have had opportunity to escalate the attack with unknown consequences.
The company acknowledges in its report that “a security incident is a systems failure.” The attack demonstrates the importance of not including credentials in source code and of minimizing the privileges of DevOps systems. “There is never a reason to trust a VCS or CI system with high value secrets. They should never ever need any power beyond running tests, accessing a test environment, or sending notifications,” said a security professional on Hacker News, who also questioned the security design of CircleCI but added that “all its competitors are just as bad.”