GitHub Outage Highlights Capacity Management Shortcomings
On August 17, GitHub experienced a widespread outage lasting 7 hours and 47 minutes, disrupting users during work hours. The incident impacted GitHub.com, APIs, Git operations, as well as Issues, Pull Requests, Actions, and Copilot services. At the peak of the disruption, approximately 20% of web and API requests failed, with failures climbing to nearly 50% for downloading archives and raw content.
User Impact and Service Disruptions
For many users, the outage resulted in intermittent connection failures and challenges in accessing repositories. Tasks such as code submission and retrieval, managing Issues and Pull Requests, executing GitHub Actions, and accessing Copilot became unreliable. Teams were unable to rely on GitHub for synchronizing code, conducting reviews, merging changes, or executing their integration and deployment pipelines.
Scaling Issues Unearthed
In a post on its website, GitHub attributed the outage to an unprecedented surge in traffic, exacerbated by a flaw in how capacity was managed. The auto-scaling feature, designed to automatically allocate resources based on observed load, was misconfigured. It monitored the host service within its Central US data center but failed to account for the limiting factor introduced by an adjacent network component.
The Role of the Sidecar Configuration
Specifically, the issue revolved around an Istio sidecar, a container that facilitates service networking without performing business logic. This sidecar reached its concurrency limit—the maximum number of simultaneous requests it could handle—however, the automated scaling rules did not trigger additional resources when it became saturated.
The saturation of the Istio sidecar extended to four HAProxy nodes, which are responsible for traffic distribution across multiple servers. These nodes quickly ran out of available connections, leading to authentication delays and failures in dependent services. GitHub ultimately restored service by simultaneously suspending HAProxy on these nodes and diverting traffic to Northern Virginia.
Additional Challenges from HTTP Retry Mechanisms
HTTP retries during the outage further strained the load balancers. GitHub’s internal configuration, described as “optimistic,” increased the volume of requests directed at already burdened systems. Notably, the Copilot token service experienced overwhelming demand due to delayed responses from an internal gateway, which triggered a cascading failure in Visual Studio Code (VS Code). What normally generated 7,000 to 9,000 requests per second skyrocketed to between 70,000 and 100,000 requests per second.
This ten-fold increase, based on GitHub’s own data, accounts for the slower recovery of Copilot, although not for the entire outage. Most services resumed by 4:36 PM UTC, while GitHub Actions remained degraded until approximately 6:03 PM, and the Copilot token service only fully recovered by 9:02 PM after curtailing excessive retries and temporarily blocking responses that perpetuated the loop.
The Azure Transition: A Double-Edged Sword
The outage comes amid GitHub’s rapid infrastructure transformation, with monthly commits soaring from 1.4 billion in April to 2.9 billion. Azure now handles about 58% of the platform’s traffic and half of Git operations, a significant increase from just 12% in May.
GitHub has positioned its migration to Azure as a strategy to lessen reliance on its own data centers and shared infrastructures. However, this transition does not serve as the root cause of the outage. As traffic was being rapidly shifted to Azure, common components retained sufficient importance to propagate failures across authentication, APIs, and development tools.
This was not the first incident during the transformation; a previous outage related to GitHub Actions on August 6 had already raised alarms, attributed to resource shortages and cascading failures within a service still predominantly operated from their data centers. The recurring theme of capacity-related incidents within a single month suggests that the practices in place to identify limitations are struggling to keep pace with increased usage. Vlad Fedorov, GitHub’s CTO, acknowledged this discrepancy in the company’s assessment on August 20.
Path Forward: Addressing Capacity and Failures
In response, GitHub has committed to revising its auto-scaling rules, auditing Istio limits, adjusting retry budgets and timeouts, and enhancing monitoring of load balancers and regional failovers. The company also aims to rectify the behavior of VS Code, which contributed to the traffic surge for Copilot.
The new retry budgets will set a maximum number of allowed retries before abandoning a request entirely. These adjustments, along with variable wait times and the cessation of cascading requests, will soon be treated as capacity parameters rather than mere reliability settings. While the next traffic peak might not overload the same component, a retry will only be beneficial if the target service is still capable of responding.
Nonetheless, GitHub’s technical report remains the sole source detailing the causal chain behind the incident. The company refrained from naming the affected host service, did not specify traffic distribution between the initial peak, retry attempts, and automated collection processes, and offered no timeline for implementing the announced changes. This incident serves as a critical reminder for enterprises: although infrastructure may expand, the observability and resilience mechanisms need to evolve at a matching pace.



