Microsoft Reveals Hidden Bug Causing Global Outlook and Microsoft 365 Outage
More than 80% of servers were unresponsive at the worst point after a flaw introduced in July went undetected. Microsoft reveals the details behind the widespread outage that affected Outlook email, calendars, meeting invitations, and other Microsoft 365 tools.
Microsoft 365 Executive Charles Lamanna explains that a hidden software bug, stemming from a fix developed for gateway servers, triggered a chain reaction during peak U.S. business hours on November 12th. The bug, combined with increased demand and reduced server capacity, caused a global standstill.
Root Cause
According to Lamanna, three factors contributed to the outage:
- A latent defect: Servers became overwhelmed under certain load conditions.
- Increased demand: Traffic shifted to a system supporting Microsoft Teams and email services.
- Reduced server capacity: Approximately 14% of the affected system's servers were already offline due to a separate issue.
The bug was unique as it had been operating worldwide for over three months without issues, passing through Microsoft's testing due to a mismatch between the traffic and hardware conditions in the lab and those that triggered the problem.
Recovery and Impact
Recovery was slow due to servers being overwhelmed with traffic upon coming back online. The automated recovery systems were not equipped to handle such a widespread breakdown, and the bug could not be quickly rectified using normal rollback tools.
Engineers eventually applied a fix and restarted groups of servers, gradually restoring service. However, the outage affected Microsoft's Copilot AI assistant and customer support systems for airlines like IndiGo and Akasa Air, as well as OpenAI and Anthropic's integrations with Microsoft 365.
Rep. Rich McCormick, R-Ga., highlighted the vulnerability of digital systems used by businesses, hospitals, schools, and government agencies, warning about potential consequences if malicious actors were behind such disruptions.
He referenced his Common-Sense Cloud Security Act, emphasizing the need for improved risk assessment, threat detection, and damage containment in cloud environments.