All Systems Operational

Website




Operational

Video playback




Operational

Search




Operational

Recording / Uploading




Operational

Processing / Encoding




Operational

Integrations




Operational

External Services

AWS CloudFront

AWS EC2

AWS Route 53

AWS S3

Amazon Web Services

SendGrid

WebEx

Zoom

Locations

History (Last 7 days)

Panopto NA, EU, AP, CA, and AU Cloud Notification 2022-05-19Operational

Incident Status

Operational


Components

Integrations


Locations

North America (NA), Europe (EU), Asia-Pacific (AP), Canada (CA), Australia (AU)



May 19, 2022 8:10PM UTC
[Resolved] From 2022-05-19 16:30 UTC to 2022-05-19 19:40 UTC, the Panopto NA, EU, AP, CA, and AU Clouds experienced increased error rates in LMS integrations. During this time, users would have been unable to add new videos or folders to their LMS courses. We have resolved this incident and Panopto engineers are completing a root cause analysis, which we will post here, within the next week.

May 20, 2022 6:53PM UTC
[Identified] We have discovered that some Panopto folders created during the outage do not properly allow end user's access via LMSs. Our team is working on fixing these impacted folders. Our next update will be within 3 hours.

May 20, 2022 10:00PM UTC
[Identified] Our team is still working on fixing these impacted folders. Our next update will be within 3 hours.

May 21, 2022 12:46AM UTC
[Resolved] Our team has fixed the impacted folders, and end users should now be able to properly access those folders via their LMS. We have resolved this incident and Panopto engineers are completing a root cause analysis, which we will post here, within the next week.
Panopto NA and EU Cloud Notification 2022-05-10Operational

Incident Status

Operational


Components

Integrations


Locations

North America (NA), Europe (EU)



May 10, 2022 8:38AM UTC
[Monitoring] The Panopto NA and EU Clouds have experienced an increased error rate in Canvas LTI. We’re sorry for this interruption. Our cloud operations team has mitigated the issue. Our cloud operations team is monitoring cloud metrics to ensure the issue is resolved. Our next update will be within 30 minutes.

May 10, 2022 9:02AM UTC
[Resolved] The Panopto NA and EU Clouds have been stable and metrics have remained normal since our last update. We are resolving this incident and Panopto engineers are completing a full root cause analysis, which we will post here, within the next week.

May 19, 2022 11:05PM UTC
[Resolved] On 2022-05-06 UTC, we deployed a service update to the Panopto AU and AP clouds which contained a bug in LMS course structure provisioning. This bug prevented all provisioning operations, including LTI folder links, from working in some configurations. We resolved the issue with a settings change at 2022-05-07 20:14 UTC. On 2022-05-09 UTC, we deployed the same service update to the Panopto NA, EU, and CA clouds. Due to human error during those deployments, the settings changes required to prevent the bug were not applied. We resolved the issue with a settings change at 2022-05-10 02:02 UTC. RCA: The Panopto 12.8.0 release contained a flaw in a change to LMS course structure provisioning. This change did not properly account for common course structure copy settings. Our team failed to detect this issue due to insufficient test coverage. To prevent a recurrence of this issue, we will audit common combinations of integration settings and expand our automated and manual test cases to resolve identified testing gaps. Additionally, we will improve our deployment process to automate the incorrectly executed manual process.
Panopto AU Cloud Notification 2020-05-09Degraded Performance

Incident Status

Degraded Performance


Components

Processing / Encoding


Locations

Australia (AU)



May 9, 2022 8:42AM UTC
[Investigating] The Panopto AU Cloud is experiencing a performance degradation in video processing. Our cloud operations team is investigating the issue and we’ll update here as soon as we know more. Our next update will be within 30 minutes. What does this mean? Content processing or other long-running operations may take an extended amount of time to complete.

May 9, 2022 9:07AM UTC
[Identified] The source of the earlier performance degradation has been mitigated. We expect continued delays for the next 60 minutes while our system processes through the backlog. Our cloud operations team is monitoring performance metrics to ensure the issue is resolved.

May 9, 2022 10:15AM UTC
[Resolved] The Panopto AP Cloud has been stable and metrics have remained normal since our last update. We are resolving this incident and Panopto engineers are completing a full root cause analysis, which we will post here, within the next week.

May 18, 2022 1:59AM UTC
[Resolved] On 2022-05-06, we deployed a service update to the Panopto AU and AP clouds which contained a bug in Panopto’s content processing system. Under certain conditions, this bug greatly delayed content from being published. We resolved this issue with a service update at 2022-05-09 9:54 UTC. RCA: Our engineering team introduced this bug while making changes to the content processing system. In specific conditions, this change caused jobs to take considerably longer to complete processing. Due to these delayed jobs slowly building up, eventually all jobs were delayed due to a lack of available processing machines. Our team failed to detect this issue due to insufficient test coverage. To prevent a recurrence of this issue, we will expand automated testing of our encoding system to cover the identified testing gap. Additionally, we will make improvements to our alarming to better detect such processing delays.