[Monitoring] We're investigating increased processing latency in the Panopto EU Cloud.
Our cloud operations team is investigating the issue.
June 5, 2020 12:38AM UTC
[Monitoring] The source of the earlier performance degradation has been mitigated. Our cloud operations team is monitoring performance metrics to ensure the issue is resolved.
June 5, 2020 7:56PM UTC
[Resolved] Between 2020-06-04 23:14 UTC and 2020-06-05 00:15 UTC, the Panopto EU cloud experienced increased job processing latency. Our cloud operations team resolved the issue.
RCA: During this time, a deployment to the EU cloud resulted in a mismatch between actual cloud infrastructure and a configuration in the database. The cloud operations team identified the source of the mismatch and manually corrected it. The Panopto development team will make a code change to prevent the possibility of a configuration mismatch like this one from recurring.
[Investigating] We are investigating increased delays in search indexing on the NA cloud.
June 5, 2020 1:35AM UTC
[Identified] The source of the increased latency for search indexing has been identified. Our cloud operations team is working to resolve the issue.
June 5, 2020 3:31AM UTC
[Monitoring] The source of the elevated search indexing delays has been mitigated. We expect search indexing to be current at 04:00 UTC.
June 5, 2020 3:47AM UTC
[Monitoring] Search indexing became current at 2020-06-05 03:44 UTC. Our cloud operations team will continue to monitor search indexing metrics to ensure the issue is resolved.
June 5, 2020 6:10PM UTC
[Resolved] Between 2020-06-04 23:54 and 2020-06-05 03:44 UTC, the Panopto NA Cloud experienced degraded search indexing performance. Our cloud operations team has resolved the issue.
RCA: During this time, a manual operation was run which was expected to be unconnected to search indexing. However, the workload triggered an unexpectedly large search indexing workload within the Panopto NA Cloud system. A manual stop was put on the offending workload, after which time the system resumed normal functioning. The Panopto Operations team will modify the internal process that resulted in the workload. Additionally, the Panopto Operations team will modify the behavior of the search indexing system to avoid the increased load on the indexing system for this category of updates.