August 1, 2022 3:32AM UTC
[Investigating] The Panopto NA Cloud is experiencing an increased error rate in search.
We’re sorry for this interruption. Our cloud operations team is urgently investigating the issue and we’ll update here as soon as we know more.
Our next update will be within 30 minutes.
What does this mean?
Searching for content may fail to succeed.
August 1, 2022 4:04AM UTC
[Monitoring] The source of the increased error rates has been mitigated. Our cloud operations team is monitoring cloud metrics to ensure the issue is resolved.
Our next update will be within 60 minutes.
August 1, 2022 4:55AM UTC
[Resolved] The Panopto NA Cloud has been stable and metrics have remained normal since our last update.
We are resolving this incident and Panopto engineers are completing a root cause analysis, which we will post here, within the next week.
August 4, 2022 7:09PM UTC
[Resolved] On 2022-08-01 from 3:20 to 3:46 UTC, the Panopto NA Cloud experienced increased error rates in search. During this time, customers will have experienced errors when searching for content.
RCA: Due to an issue in our search service, some search servers began to slowly consume excessive resources over time. Eventually, the servers became overloaded and began failing to serve requests. Upon being alerted to this, we resolved the issue by fully resetting the impacted servers.
To prevent a recurrence of this issue, we will fix this excessive resource consumption issue in our search service. Additionally, we will add additional alarms to detect such excessive resource consumption before it causes issues in the future.