GitHub is to rate limit the audit log available to users of its Enterprise Cloud, the premium tier for customers using GitHub.com as opposed to self-hosting. The limit will be 15 queries per minute from August 1, 2023, with further limiting possible in future.
The audit log is a critical feature for detecting security issues, monitoring policy compliance, and investigating incidents. It records activities such as permission changes and user account actions, codespace creation and deletion, repository cloning, IP allow list changes, SSH key changes, repository visibility changes, and much more. Logs for events other than git events are retained for six months, and for git, seven days. The idea is to give organizations real-time monitoring of GitHub activity.
It appears though that some users have been getting too close to real-time monitoring, or have too many different processes accessing the audit logs. GitHub says that “query cost is a crucial consideration, and in the future, the audit log may impose further rate limiting for high-cost queries that place significant strain on our data stores.”
The suggested actions for customers are to get ready to handle HTTP 429 responses – meaning “too many requests” – or, if “near real-time data” is a requirement, to stream the audit log to another service and then query that service instead – meaning that the customer, rather than GitHub, will pay for resource usage other than the stream itself. Audit log streaming supports Amazon S3, Azure Blob Storage or Event Hub, Datadog, Splunk, or Google Cloud storage.
What does a GitHub rate limit mean? The answer is complicated. There are always rate limits for API calls and there is even an API for users to get their rate limit status, for which GitHub kindly notes that “accessing this endpoint does not count against your REST API rate limit.” The GraphQL API is problematic because “A single complex GraphQL call could be the equivalent of thousands of REST requests,” GitHub states, and it has a points system in place to calculate the cost of queries in order to apply limits. Another API subject to special limits is Search Code, thanks to what GitHub docs say is “the complexity of searching code.”
Rate limits are annoying but it is important to keep GitHub responsive to its millions of users. That said, with GitHub it is outages rather than rate limits that are the biggest problem. “There is at least one major incident/outage with GitHub every single month and most of the time there is more than one incident,” said one developer on Hacker News, arguing that “now is the perfect time to consider self-hosting.”
The last availability report confessed that “in May, we experienced four incidents that resulted in degraded performance across GitHub services.” These are blamed on a variety of causes including “connection pool exhaustion on our proxy layer”, “a bug in the database version we are running,” “a configuration change to the internal service serving Git data,” and that “the primary database of a cluster supporting key-value data had experienced a hardware crash.”
With 94 million developers – as reported last year – GitHub has unusually tough challenges in maintaining a highly responsive platform. If rate-limiting is part of the solution, developers may feel it is a worthwhile trade.