Partial Service Disruption
February 16, 2021 1:44AM UTC
[Investigating] The Panopto EU Cloud is experiencing an increased error rate in search.
Our cloud operations team is urgently investigating the issue and we’ll update here as soon as we know more.
Our next update will be within 30 minutes.
What does this mean?
The affected components are degraded to the point of failure so they are no longer functional. This indicates that the affected aspects of the Panopto platform are currently unavailable to users.
February 16, 2021 2:13AM UTC
[Investigating] The Panopto EU Cloud is continuing to experience an increased error rate in search.
Our engineering team is actively investigating this issue with urgent priority. Please bear with us while we continue to investigate the cause.
Our next update will be within 30 minutes.
February 16, 2021 2:43AM UTC
[Identified] The source of the increased error rates has been identified. Our cloud operations team is applying a mitigation for the issue.
Our next update will be within 30 minutes.
February 16, 2021 3:00AM UTC
[Monitoring] The source of the increased error rates has been mitigated. Our cloud operations team is monitoring cloud metrics to ensure the issue is resolved.
Our next update will be within 60 minutes.
February 16, 2021 4:07AM UTC
[Resolved] The Panopto EU Cloud has been stable and metrics have remained normal since our last update.
We are resolving this incident and Panopto engineers are completing a root cause analysis, which we will post here, within the next week.
February 24, 2021 4:24PM UTC
[Resolved] Panopto EU Cloud Notification 02-16-2021 - increased search error rates
On 2021-02-16 from 01:09 UTC to 02:50 UTC, the Panopto EU Cloud experienced increased latencies and error rates on search indexing and queries.
RCA: During an automated search cluster cleanup operation, a search index cache began underperforming due to the increased load from the cleanup process. This underperformance caused some updates to the search index to fail, and search queries failing or taking unusually long.
To prevent a recurrence of this issue, our engineering team has removed the reliance on the underperforming cache as performance testing showed this cache was not necessary for continued high-performance operation. Our engineering team will investigate and test for other potential similar bottlenecks and areas where shared caches are used.
