September 19, 2023 7:56PM UTC
[Resolved] On 2023-09-08 from 03:50 UTC to 2023-09-09 00:44 UTC, the Panopto NA Cloud experienced an issue with SSO logins from the Blackboard integrations that are using the new Panopto integration for Blackboard Original courses. During this time, customers will have experienced an HTTP 500 error when attempting to log into Panopto through their Blackboard integration.
RCA: The root cause of this issue was an improperly configured test server that was used while Panopto was verifying a fix for an issue with Blackboard Permission Syncing, which was released with Panopto 14.3 on 2023-09-08. Due to the test server being improperly configured, our testing missed this key regression for customers who had already migrated to the new Blackboard integration, which led to the login error after Panopto 14.3 was released.
To prevent a recurrence of this issue, Panopto will be adding additional automation testing for the new Blackboard Integration for Blackboard Original courses to catch errors on basic login operations. In addition, we will be verifying all integration configurations in our test environments to ensure additional confidence in our testing before releasing bug fixes and new features related to our integrations.
September 9, 2023 6:43AM UTC
[Resolved] The Panopto NA Cloud has been stable and metrics have remained normal since our last update.
We are resolving this incident and Panopto engineers are completing a root cause analysis, which we will post here, within the next week.
September 8, 2023 9:13PM UTC
[Identified] Our cloud operations team is still on track to deploy a fix for this issue tonight.
Our next update will occur once the deployment concludes and the fix is verified.
September 8, 2023 4:48PM UTC
[Identified] The Panopto NA Cloud is experiencing an increased error rate for sign-in for a subset of customers using the Blackboard integration.
Our cloud operations team is building and testing a fix which we aim to deploy tonight. We have also come up with a temporary mitigation which may negatively impact LTI 1.1 quiz embeds. Due to the side effects of the mitigation, please contact Panopto Support to discuss if applying this workaround will work for your site.
Our next update will be within the next 4 hours.
September 8, 2023 4:06PM UTC
[Identified] The source of the increased error rates has been identified. Our cloud operations team is working on a mitigation for the issue.
Our next update will be within 30 minutes.
September 8, 2023 3:36PM UTC
[Investigating] The Panopto NA Cloud is experiencing an increased error rate for sign-in for a subset of customers using the Blackboard integration.
Our cloud operations team is urgently investigating the issue and we’ll update here as soon as we know more.
Our next update will be within 30 minutes.
What does this mean?
Some users will be unable to sign-in to Panopto via the Blackboard provider
