Resolved Major Service Disruption

On Thursday, February 2, 2017, between 4:16am and 6:20am PST, Okta experienced a service disruption in US Cell 2 in which end-users experienced latency and errors while accessing Office 365. Between 4:16am and 5:05am PST end-users in US Cell 2 experienced an average 25% error rate connecting to Office 365. Error rates increased between 5:06am and 6:20am PST, and peaked as high as 95% at 6:01am PST. During this time, users may also have experienced latency or errors when connecting to other applications. From 5:30am and 6:05am, end-users in US Cell 1 - 4 also experienced sporadic latency and timeouts, with latency peaking at 40%, while error and timeouts remained below 10% of requests for interactive user sessions. This degradation was mitigated and service returned to normal at approximately 6:20am PST. The service disruption was triggered by an anomalous spike in Office 365 authentication requests which were not correctly rate-limited. Okta is investigating the source of the spike in Office 365 authentication requests. Okta has also corrected the rate-limit configuration and added additional capactiy and tuning improvements to prevent this issue from occurring in the future. Event Root Cause Analysis details can be found [here](https://support.okta.com/help/articles/Knowledge_Article/Root-Cause-Analysis-Service-Disruption-02022017).