Active Incident
Panopto Cloud Notification 2021-10-15 - Zoom IntegrationPartial Service Disruption

Incident Status

Partial Service Disruption

Components

Integrations

Locations

North America (NA), Europe (EU), Asia-Pacific (AP), Canada (CA), Australia (AU)


October 15, 2021 8:07PM UTC
[Identified] Panopto's Engineering team has identified an issue with the Zoom integration where we are failing to delete recordings from Zoom Cloud after we have imported them to Panopto. This affects customers who have the “Delete from Zoom after successful import” setting enabled in Panopto’s Zoom Integration. The issue began at 2021-10-06 10:42:00 UTC. Panopto's Engineering team has identified the cause of the issue and is currently processing the backlog of deletions. The deletions from Zoom Cloud may be slower than usual for the next week. We will update this post when we have fully resolved the issue.

Website




Operational

Video playback




Operational

Search




Operational

Recording / Uploading




Operational

Processing / Encoding




Operational

Integrations




Partial Service Disruption

External Services

AWS CloudFront

AWS EC2

AWS Route 53

AWS S3

Amazon Web Services

SendGrid

WebEx

Zoom

Locations

History (Last 7 days)

Panopto EU Cloud Notification 2021-10-11Operational

Incident Status

Operational


Components

Video playback


Locations

Europe (EU)



October 11, 2021 5:27PM UTC
[Investigating] On 2021-10-11 between 13:45 - 14:00 UTC and 14:26 - 14:29 UTC, we noticed elevated latency in our webcasting services in Panopto EU cloud. During these time periods, the customer might have experienced lagging or dropped connections during webcasting. We apologize for the inconvenience! Our cloud ops team has mitigated the webcast issue, so the service has already been restored. We are actively investigating the root cause and will post here within the next week.

October 11, 2021 5:29PM UTC
[Resolved] The webcasting service has already been restored. We are actively investigating the root cause and will post here within the next week.

October 15, 2021 11:09PM UTC
[Resolved] On 2021-10-11 from 13:45 UTC to 14:00 UTC and from 14:26 UTC to 14:29 UTC, the Panopto EU Cloud experienced increased latencies and error rates in RTMP webcasts. During this time, customers saw buffering and streaming disconnections. RCA: A data store service provided by our cloud hosted provider showed periodic highly elevated latencies for a percentage of write operations. Due to the large number of write operations needed and the occasional high latencies encountered, the servers that handle RTMP broadcasts became backlogged cascading the issue. This backlogging caused delays in publishing of video data and disconnections. The Panopto operations team mitigated the issue by increasing the number of servers available until system performance stabilized. To prevent a recurrence of this issue, the Panopto operations team will improve the resiliency of the RTMP servers data store operations to better isolate and automatically mitigate latency spikes.
Panopto NA Cloud Notification 2021-10-07Degraded Performance

Incident Status

Degraded Performance


Components

Website


Locations

North America (NA)



October 7, 2021 2:44PM UTC
[Investigating] The Panopto NA Cloud is currently experiencing an increased error rate in video permission management. Our cloud operations team is actively investigating this issue. We're sorry for this interruption. Our next update will be within 30 minutes. What does this mean? Attempts to share or update video permissions may fail.

October 7, 2021 2:56PM UTC
[Monitoring] The issue has now been mitigated. Our cloud operations team is monitoring cloud metrics to ensure the issue is resolved. Our next update will be within 1 hour.

October 7, 2021 3:44PM UTC
[Resolved] The Panopto NA Cloud has been stable and metrics have remained normal since our last update. We are resolving this incident and Panopto engineers are completing a root cause analysis, which we will post here, within the next week.

October 15, 2021 4:40AM UTC
[Resolved] On 2021-10-07 from 11:30 UTC to 14:55 UTC, the Panopto NA Cloud experienced increased error rates in sharing and permissions management. During this time, customers saw slow performance or errors while attempting to share content or modify permissions. RCA: The Panopto 11.7.0 release mistakenly enabled a maintenance job which issues long-running operations on the data store that holds content permissions. These long-running operations blocked updates from customers using the product, resulting in slow performance or errors. We enabled this maintenance job due to human error in a previous cloud update and in preparing this cloud release. The Panopto operations team resolved the issue by disabling this maintenance job. To prevent a recurrence of this issue, the Panopto operations team will improve outage response playbooks and processes, and we will refresh the team on best practices to protect against human error in the release process.