The world’s self-declared “leading software development platform” wasn’t this morning, as GitHub spluttered and died as developers in Asia were just starting their week and continued to rest as Europe started turning on.
As of 10:30 BST, developers were still reporting 404s when they tried to view pull requests.
Things started going awry shortly after midnight BST as the code repo platform reported: “We are investigating reports of elevated error rates.”
Four minutes later, at 00:13, it said: “We are investigating reports of service unavailability.” Half an hour later, it added, “We’re investigating problems accessing GitHub.com.” At 01:43, it said “We’re failing over a data storage system in order to restore access to GitHub.com.”
After that, developers who could no longer see their code, could at least see a message repeated every twenty minutes, saying: “We are continuing to work to migrate a data storage system in order to restore access to GitHub.com.”
As of 9:19 BST, GitHub was telling frustrated developers it expected to be restored within the hour, folowing that up with a 10:47 admission that “We continue to monitor restores which are taking longer than anticipated. We estimate they will be caught up in an hour and a half.”
The official incident report at time of writing said: “At 10:52 pm Sunday UTC, multiple services on GitHub.com were affected by a network partition and subsequent database failure resulting in inconsistent information being presented on our website.
“Out of an abundance of caution we have taken steps to ensure the integrity of your data, including pausing webhook events and other internal processing systems.”
It added, “no data was lost” and the “incident…only impacted website metadata stored in our MySQL databases, such as issues and pull requests.”
The fail comes at a sensitive time for GitHub. Just last week it sought to extend its reach into container workflow automation with its GitHub Actions product. That was quickly followed by the EU’s thumbs up to its proposed takeover by Microsoft.
Frustrated devs referenced the Microsoft takeover this morning, though the actual deal is nowhere near concluded.
Darn Microsoft products… #github https://t.co/9hZAEPXJ5L
— Daniel (@DanielRam86) October 22, 2018
Other developers took a sanguine approach.
#github PRs are not still going through https://t.co/uPHRA5OILj. Time to drink more coffee and take a chill pill ☕️
— Toni Dahl (@k1ll3rf0x) October 22, 2018