Partial Service Disruption
North America (NA), Europe (EU), Canada (CA)
September 11, 2024 7:29PM UTC
[Resolved] RCA: Our Engineering team made a change in the 15.2.0 release that involved modifying how we pass settings and LMS configuration information between our various infrastructures. As part of that change, a bug was introduced that resulted in Blackboard Identity Providers that had been migrated from the Blackboard Building Block to our new Blackboard Ultra integration receiving the wrong login URL for use in authenticating a user with Blackboard. This resulted in users being unable to successfully log into Panopto through those Blackboard providers.
To prevent a recurrence of this issue, our Engineering team will add additional automated testing specific to customers who have migrated a Blackboard Building Block integration to a Blackboard Ultra integration in order to avoid missing similar issues in the future.
September 10, 2024 4:29PM UTC
[Monitoring] The system has been operating normally the few days. We will be posting a root cause analysis (RCA) in the next two days.
September 9, 2024 4:08PM UTC
[Monitoring] The system has been operating normally the last couple of days. We will continue to monitor to ensure system stability.
Our next update will be within 24 hours.
September 8, 2024 2:48AM UTC
[Monitoring] The system is operating normally. We will continue to monitor to ensure system stability.
Our next update will be within 24 hours.
September 7, 2024 3:37AM UTC
[Monitoring] Deployment of the mitigation to all Panopto Cloud environments is complete. We will continue to monitor to ensure system stability.
Our next update will be within 24 hours.
September 6, 2024 11:43PM UTC
[Identified] The source of the increased error rates has been identified. Our cloud operations team has verified the mitigation for the issue, and deployment to the Production environment will begin shortly.
We will update again after deployment is complete.
September 6, 2024 8:56PM UTC
[Identified] The source of the increased error rates has been identified. Our cloud operations team is preparing the mitigation for the issue.
Our next update will be within 3 hours.
September 6, 2024 6:57PM UTC
[Identified] The source of the increased error rates has been identified. Our cloud operations team is working on a mitigation for the issue.
Our next update will be within 2 hours.
September 6, 2024 5:52PM UTC
[Identified] The source of the increased error rates has been identified. Our cloud operations team is continuing to work on a mitigation for the issue.
Our next update will be within 60 minutes.
September 6, 2024 4:54PM UTC
[Identified] The source of the increased error rates has been identified. Our cloud operations team is working on a mitigation for the issue.
Our next update will be within 60 minutes.
September 6, 2024 3:50PM UTC
[Identified] The source of the increased error rates has been identified. Our cloud operations team is investigating mitigation paths for the issue.
Our next update will be within 60 minutes.
September 6, 2024 2:58PM UTC
[Investigating] The Panopto NA , EU & CA Clouds are experiencing increased error rates when signing in using the Blackboard Original login provider.
Our cloud operations team is urgently investigating the issue and we'll update here as soon as we know more.
Our next update will be within 60 minutes.
What does this mean?
Users navigating to their Panopto site and attempting to sign in via the Blackboard Original integration will be unable to do so. This is true for the web interface and the Mobile/Recording clients.
What can be done to mitigate this issue?
As other sign-in providers are unaffected (including Blackboard Ultra), users can either select another available identity provider, or the local Panopto Administrator at your institution can update the "Default Sign-in Option" to True on an alternative provider that users have access to, such as Blackboard Ultra.
