All Systems Operational

Website




Operational

Video playback




Operational

Search




Operational

Recording / Uploading




Operational

Processing / Encoding




Operational

Integrations




Operational

External Services

Locations

History (Last 7 days)

Incident Status

Operational


Components

Website, Video playback, Search, Recording / Uploading, Processing / Encoding, Integrations


Locations

North America (NA)




November 22, 2021 6:20PM UTC
[Monitoring] On 2021-11-22 from 18:00 UTC to 18:05 UTC, the Panopto NA Cloud experienced a partial service outage. During this time, some customers will have experienced errors in using Panopto NA Cloud. The issue has already been mitigated. We will continue to monitor the situation to make sure the issue is fully recovered.

November 22, 2021 8:51PM UTC
[Resolved] The Panopto NA Cloud has been stable and metrics have remained normal since our last update. We are resolving this incident and Panopto engineers are completing a root cause analysis, which we will post here, within the next week.

December 4, 2021 1:43AM UTC
[Resolved] On 2021-11-22 from 18:03 UTC to 18:09 UTC, the Panopto NA Cloud experienced a partial service outage. During this time, a subset of customers were unable to access Panopto. RCA: Panopto’s operation team caused this outage due to a mistake in responding to an unusual pattern of network traffic to our cloud servers. An increase of unusual network traffic caused alarms to fire. The team identified that the traffic originated from a single IP address which needed to be temporarily restricted from accessing Panopto’s services. This traffic passed through a Panopto-operated software load balancer that supports multiple customers, and the Panopto operations team accidentally restricted all traffic from that software load balancer, which prevented those customers from accessing Panopto. Panopto’s operations team identified the error when additional alarms fired and repaired the issue, restoring access for all affected customers. To prevent a recurrence of this issue, our operations team has updated the playbook for this type of issue to ensure that there are specific steps that will prevent our engineers from making this mistake in the future.