Active Incident

Incident Status

Operational

Components

Search

Locations

North America (NA)



September 25, 2020 1:22AM UTC
[Investigating] We are investigating elevated error rates in serving search results and increased latency in search indexing in the NA cloud. Our cloud operations team is investigating the issue.

September 25, 2020 2:52AM UTC
[Monitoring] The source of the increased error rates has been mitigated. Our cloud operations team is monitoring performance metrics to ensure the issue is resolved.

Website




Operational

Video playback




Operational

Search




Operational

Recording / Uploading




Operational

Processing / Encoding




Operational

Integrations




Operational

External Services

Locations

History (Last 7 days)

Incident Status

Degraded Performance


Components

Search


Locations

North America (NA)




September 22, 2020 9:49PM UTC
[Investigating] We are investigating elevated error rates in search indexing. Our cloud operations team is investigating the issue.

September 23, 2020 5:32AM UTC
[Monitoring] The source of the increased error rates has been mitigated. Our cloud operations team is monitoring performance metrics to ensure the issue is resolved.

September 25, 2020 11:56PM UTC
[Resolved] On 2020-09-22 between 21:01 and 5:12 UTC, the Panopto NA Cloud experienced increased error rates in search indexing. Our cloud operations team has resolved the issue. RCA: Beginning at 21:00, an emergent usage pattern lead to a large amount of indexing work being generated. The cloud operations team identified that this work was ultimately throttled by a bottleneck in the indexing system. To resolve the issue, the operations team took action to mitigate via a manual scaling action. To prevent a recurrence of this issue, our engineering team will update the system to remove this bottleneck and implement performance improvements for the observed usage pattern.

Incident Status

Service Disruption


Components

Website, Video playback, Search, Recording / Uploading, Processing / Encoding, Integrations


Locations

North America (NA)




September 17, 2020 6:51PM UTC
[Investigating] We are investigating increased error rates on requests to the Panopto NA Cloud. Our cloud operations team is investigating the issue.

September 17, 2020 7:00PM UTC
[Monitoring] The source of the increased error rates has been mitigated. Our cloud operations team is monitoring performance metrics to ensure the issue is resolved.

September 17, 2020 7:50PM UTC
[Monitoring] The source of the increased error rates on requests has been mitigated. Our cloud operations team is working on mitigating related increased delays in search indexing on the NA cloud.

September 17, 2020 8:59PM UTC
[Monitoring] The delays in search indexing have been mitigated. Our cloud operations team is monitoring performance metrics to ensure the issue is fully resolved.

September 19, 2020 3:22AM UTC
[Resolved] On 2020-09-17 between 18:10 and 18:45 UTC, the Panopto NA Cloud experienced increased error rates. Between 18:45 and 20:59 UTC, the Panopto NA Cloud experienced increased latencies in search indexing. Our cloud operations team has resolved the issues. RCA: An unusual, rapid spike of new recordings resulted in blocking in one of the databases in Panopto's NA Cloud. The spike of recordings originated from a misconfigured Panopto Remote Recorder. The database blocking initially caused increased error rates and slow performance to the NA Cloud. Subsequently, a spike of search indexing jobs caused a delay in search indexing. To prevent a recurrence of this issue, our engineering team will implement both client-side and server-side throttling on recorder stream creation.

September 24, 2020 7:49PM UTC
[Resolved] On 2020-09-22 between 21:01 and 5:12 UTC, the Panopto NA Cloud experienced increased error rates in search indexing. Our cloud operations team has resolved the issue. RCA: Beginning at 21:00, an emergent usage pattern lead to a large amount of indexing work being generated. The cloud operations team identified that this work was ultimately throttled by a bottleneck in the indexing system. To resolve the issue, the operations team took action to mitigate via a manual scaling action. To prevent a recurrence of this issue, our engineering team will update the system to remove this bottleneck and implement performance improvements for the observed usage pattern.

Description

The Panopto operations team is performing database maintenance in the EU Cloud.


Components

Website, Video playback


Locations

Europe (EU)


Schedule

September 21, 2020 8:00PM - September 21, 2020 9:00PM UTC



September 21, 2020 8:00PM UTC
[Update] This maintenance is beginning now.

September 21, 2020 8:42PM UTC
[Update] This maintenance has successfully completed.