6/18/2019
Just after midnight (EST) on Tuesday 6/18/19, #open support channels began to receive sporadic reports from users unable to access their accounts. At 6:50 a.m we made the decision to take our servers down for emergency maintenance to investigate.
After we confirmed that there was no evidence of unauthorized access to any of our infrastructure, we started reconstructing what had occurred. The cause of the issue was identified as an interaction between a bug and some specific and unusual circumstances which caused parts of some profiles to overwrite themselves with bad data.
We then got to work in tandem on two tasks: Restoring user’s profiles to their state right before the original issue, and fixing the root cause so that it couldn’t happen again. After verifying the results and the fix, service was restored at 9:00 p.m. on 6/18.
Unfortunately, by 3:00 p.m. on Wednesday (6/19) it became apparent that some profiles hadn’t been fully restored; shortly thereafter, we took the system down for another round of emergency maintenance. This round of restorations was more complex as we needed to restore the profiles that still had issues and then account for any changes that had been made since the first downtime. At 12:03 a.m. on Thursday (6/20), all services were brought back online.
