Website, Video playback, Search, Recording / Uploading, Processing / Encoding, Integrations
November 22, 2021 6:20PM UTC
[Monitoring] On 2021-11-22 from 18:00 UTC to 18:05 UTC, the Panopto NA Cloud experienced a partial service outage. During this time, some customers will have experienced errors in using Panopto NA Cloud. The issue has already been mitigated. We will continue to monitor the situation to make sure the issue is fully recovered.
November 22, 2021 8:51PM UTC
[Resolved] The Panopto NA Cloud has been stable and metrics have remained normal since our last update.
We are resolving this incident and Panopto engineers are completing a root cause analysis, which we will post here, within the next week.
December 4, 2021 1:43AM UTC
[Resolved] On 2021-11-22 from 18:03 UTC to 18:09 UTC, the Panopto NA Cloud experienced a partial service outage. During this time, a subset of customers were unable to access Panopto.
RCA: Panopto’s operation team caused this outage due to a mistake in responding to an unusual pattern of network traffic to our cloud servers. An increase of unusual network traffic caused alarms to fire. The team identified that the traffic originated from a single IP address which needed to be temporarily restricted from accessing Panopto’s services. This traffic passed through a Panopto-operated software load balancer that supports multiple customers, and the Panopto operations team accidentally restricted all traffic from that software load balancer, which prevented those customers from accessing Panopto. Panopto’s operations team identified the error when additional alarms fired and repaired the issue, restoring access for all affected customers.
To prevent a recurrence of this issue, our operations team has updated the playbook for this type of issue to ensure that there are specific steps that will prevent our engineers from making this mistake in the future.
