Microsoft’s Outage Blamed On 3 Months Hidden Bug

According to a report by Ars, a preliminary Post-Incident Report (PIR) released by the company outlines how the latent bug, changes in traffic, and reduced server capacity combined to knock much of Microsoft 365 and Exchange infrastructure offline at the end of last week. Microsoft’s report lists the incident’s final status as “a technical issue not related to cyber or AI/agents.” Microsoft’s preliminary report places the outage between November 12 at 16:20 UTC and November 13 at 16:10 UTC, just under 24 hours total, and additional monitoring remained in place for any affected systems through November 14. The outage started on November 12 and quickly rippled through businesses and public offices that depend on Microsoft’s cloud and email exchange infrastructure. The outage is covered in Microsoft’s incident IDs EX1550543 and MO1550821.
You need Microsoft 365 admin credentials to see the full report on the company’s status pages, but Office Watch has reproduced the preliminary report, and the University of Pennsylvania maintains an archive of Microsoft’s service health updates. Microsoft attributed the outage to three factors. A change deployed to Microsoft’s cloud gateway starting in July created a bug where, under certain conditions, CPU and memory resources could become exhausted and unresponsive. Microsoft contends that the defect was never discovered in pre-release testing because the hardware used was less powerful than some systems already deployed in production, where higher CPU core counts on newer machines apparently contributed to the lockout.
The second factor was server load. Microsoft said that “Traffic shifts during standard network maintenance and upgrades exposed a latent bug.” The company keeps two independent gateway systems in place to access Microsoft 365 for resilience. If one system encounters trouble and users can’t get through the front door, the other is opened to keep traffic flowing. According to the report, about 15 hours prior to the outage, traffic from Teams was shifted from the alternate gateway to the primary, where Exchange Online access is maintained.
The switchover was a standard fail-safe, but it left the gateway handling more traffic, with a different mix of requests than before. As requests continued to pile up, affected servers exhausted their capacity and refused new connection requests. The traffic shift exposed a third factor: infrastructure capacity. An existing fault meant that Microsoft’s gateway server fleet was operating at 14 percent below optimal conditions. That missing headroom could have normally absorbed the demand, but when combined with the changed traffic patterns and underlying defect, it left the gateways with lock contention as requests kept piling up. As US business operations began hitting their daily peaks, the gateway servers began to freeze.
Backup systems that shifted requests away from unresponsive servers did their job, but were met with the same lockouts on the supposedly healthy backup gateways. This created a snowball effect, leaving more than 80 percent of Microsoft’s affected gateway servers frozen in the sweep. Mechanisms in place to isolate affected systems by region seem to have failed. Microsoft’s current report doesn’t address why these isolation systems may have failed. Recovery took longer than normal because the fix required a targeted system update from Microsoft engineers.



