All Systems Operational

Website




Operational

Video playback




Operational

Search




Operational

Recording / Uploading




Operational

Processing / Encoding




Operational

Integrations




Operational

External Services

AWS CloudFront

AWS EC2

AWS Route 53

AWS S3

Amazon Web Services

SendGrid

WebEx

Zoom

Locations

History (Last 7 days)

Panopto NA Cloud Notification 2023-09-18 - Full Site OutageOperational

Incident Status

Operational


Components

Website, Video playback, Search, Recording / Uploading, Processing / Encoding, Integrations


Locations

North America (NA)



September 25, 2023 10:52PM UTC
[Resolved] On 2023-09-18 from 15:49 UTC to 15:57 UTC, customers trying to use the Panopto NA2 site were getting 5xx errors and were unable to access the site. RCA: In preparation for an upcoming maintenance, a script was generated from the database server and accidentally ran on the NA2 database. The script usually completes in a few seconds, but it lasted about 8 minutes. During this time, the application was not able to connect to NA2 primary database server. Instead, it tried to connect to the secondary NA2 database server which was not configured to be live at the time. As such, customers trying to use the site during this time encountered failures. The script causing the problem was terminated at around 15:57pm and shortly after, the health of the system was back to normal and the site was available again. To prevent a recurrence of this issue, the engineering team made sure the script used for maintenance is stored in our code repository so that we would not have to regenerate it again; thus reducing human errors. Additionally, any maintenance related work would take place only during planned maintenance period. Furthermore, Panopto engineering will implement a new and a more improved Disaster Recovery (DS) technology in the coming weeks that would make failovers automatic and complete in a few seconds.

September 18, 2023 5:27PM UTC
[Resolved] The Panopto NA Cloud has been stable and metrics have remained normal since our last update. We are resolving this incident and Panopto engineers are completing a full root cause analysis, which we will post here, within the next week.

September 18, 2023 4:13PM UTC
[Monitoring] The Panopto NA Cloud is now available for all users. The source of the outage has been mitigated. Our cloud operations team is monitoring performance metrics to ensure the issue remains resolved. Our next update will be within 60 minutes.

September 18, 2023 4:00PM UTC
[Investigating] Incident Details: We are investigating a site outage on the Panopto NA Cloud. We’re sorry for this interruption, our cloud operations team is urgently investigating the issue and we’ll update here as soon as we know more. Our next update will be within 15 minutes. What does this mean? During a site outage, end users are unable to access the service meaning they are unable to access your site, nor view, edit or upload new recordings.
Panopto NA Cloud Notification 2023-09-08Service Disruption

Incident Status

Service Disruption


Components

Integrations


Locations

North America (NA)



September 19, 2023 7:56PM UTC
[Resolved] On 2023-09-08 from 03:50 UTC to 2023-09-09 00:44 UTC, the Panopto NA Cloud experienced an issue with SSO logins from the Blackboard integrations that are using the new Panopto integration for Blackboard Original courses. During this time, customers will have experienced an HTTP 500 error when attempting to log into Panopto through their Blackboard integration. RCA: The root cause of this issue was an improperly configured test server that was used while Panopto was verifying a fix for an issue with Blackboard Permission Syncing, which was released with Panopto 14.3 on 2023-09-08. Due to the test server being improperly configured, our testing missed this key regression for customers who had already migrated to the new Blackboard integration, which led to the login error after Panopto 14.3 was released. To prevent a recurrence of this issue, Panopto will be adding additional automation testing for the new Blackboard Integration for Blackboard Original courses to catch errors on basic login operations. In addition, we will be verifying all integration configurations in our test environments to ensure additional confidence in our testing before releasing bug fixes and new features related to our integrations.

September 9, 2023 6:43AM UTC
[Resolved] The Panopto NA Cloud has been stable and metrics have remained normal since our last update. We are resolving this incident and Panopto engineers are completing a root cause analysis, which we will post here, within the next week.

September 8, 2023 9:13PM UTC
[Identified] Our cloud operations team is still on track to deploy a fix for this issue tonight. Our next update will occur once the deployment concludes and the fix is verified.

September 8, 2023 4:48PM UTC
[Identified] The Panopto NA Cloud is experiencing an increased error rate for sign-in for a subset of customers using the Blackboard integration. Our cloud operations team is building and testing a fix which we aim to deploy tonight. We have also come up with a temporary mitigation which may negatively impact LTI 1.1 quiz embeds. Due to the side effects of the mitigation, please contact Panopto Support to discuss if applying this workaround will work for your site. Our next update will be within the next 4 hours.

September 8, 2023 4:06PM UTC
[Identified] The source of the increased error rates has been identified. Our cloud operations team is working on a mitigation for the issue. Our next update will be within 30 minutes.

September 8, 2023 3:36PM UTC
[Investigating] The Panopto NA Cloud is experiencing an increased error rate for sign-in for a subset of customers using the Blackboard integration. Our cloud operations team is urgently investigating the issue and we’ll update here as soon as we know more. Our next update will be within 30 minutes. What does this mean? Some users will be unable to sign-in to Panopto via the Blackboard provider