A security researcher investigated an archive of commits on GitHub, which developers had likely thought they had deleted, uncovering secrets including a token giving admin access to all Istio repositories, widely used in the enterprise, and winning around $25K in bounties.

Researcher Sharon Brizinov was able to search public repositories on GitHub for accidental commits and search them for secrets, finding among other things a personal access token (PAT) for an Istio developer. “I analyzed this token and found it had admin access to all of Istio repositories,” he said. Istio is open source service mesh software that is widely used with Kubernetes, and if compromised could impact thousands of enterprises. The token has now been revoked.

The problem is what happens when a developer accidentally commits code that includes secrets, realizes their mistake, and attempts to remove the commit. Git repositories are designed to preserve code history, so to remove a commit is working against this; even though there are circumstances in which it is essential. A developer, for example, might hard-code a password for a quick test, or forget to exclude a file or directory containing secrets from the repository, and then make the commit before discovering the error.

In this case, how does one stop the erroneous commit from replicating to all developers fetching code from the repository? It is not a new problem; there is a question about it from 2009 on StackOverflow. The quick answer is called a force push; developers can reset the commit history to before it happened and then use git force push to make the same change on the origin repository. This works to some extent, in that it rewrites the commit history and other developers who have not already fetched the erroneous commit will not now see it, but it is not a complete solution. The bad commit remains in the GitHub cache, and can be retrieved via its hash. 

All this has been well-known for years. “The commit can still be accessible directly via SHA1. Force push does not delete the commit, it creates a new one and moves the file pointer to it. To truly delete a commit you must delete the whole repo,” said a comment back in 2013.

GitHub has a support article on the topic which includes using a utility called git-filter-repo before making a force push, but also adds that GitHub support must be contacted to remove all traces of the compromised data. Support will deference or delete any affected pull requests, run garbage collection on the server, remove cached views, and purge orphaned LFS (Large File Storage) objects.

That is a lot of work to undo an error that took only a moment, making it unsurprising that developers on occasion look for a quicker solution or are perhaps unwilling to confess an embarrassing mistake.

The problem is made worse by the existence of GH Archive, a project which records all activity on public repositories and stores it in a public dataset on Google BigQuery. This is handy for research, but also ideal for security researchers (or worse) looking for orphaned commits, which is how these deleted commits end up, if not properly processed. Brizinov used this along with a quickly coded application to grab these commits and look for secrets. “The most interesting leaked secrets were GitHub PAT tokens and AWS credentials,” he said. A common error was including a .env (environment) file within a commit, accounting for just over half of the leaks.

This is not really a GitHub flaw, though developers would be helped if there were some way of actually deleting commits without having to raise a support request.

Another option when suffering this kind of incident is to remove the value of the leaked secrets by immediately revoking tokens and changing leaked passwords.

Truffle Security offers a utility to find leaked secrets for specific organizations, but it may be too late.

GitHub suggests in its support article that developers avoid making the bad commits, by observing best practice and by scanning for sensitive data using a pre-commit hook invoking something like AWS git-secrets.