Partial Service Disruption
North America (NA), Europe (EU), Asia-Pacific (AP), Canada (CA), Australia (AU)
December 5, 2022 3:02PM UTC
[Investigating] The Panopto Cloud is experiencing an increased error rate in generating Automatic Speech Recognition results for smart search.
Our cloud operations team is urgently investigating the issue and we'll update here as soon as we know more.
Our next update will be within 30 minutes.
What does this mean?
This means that search results for spoken work will not display correctly for newly created content and "Automatic Machine Captions" will not be available for import via the Panopto editor
December 5, 2022 3:40PM UTC
[Investigating] The Panopto Cloud is continuing to experience an increased error rate in generating Automatic Speech Recognition results for smart search.
Our engineering team is actively investigating this issue with urgent priority. Please bear with us while we continue to investigate the cause.
Our next update will be within 2 hours.
December 5, 2022 6:20PM UTC
[Identified] The source of the increased error rate has been identified. Our cloud operations team is working on a mitigation for the issue.
Our next update will be within 2 hours.
December 5, 2022 8:49PM UTC
[Identified] Our Cloud Operations team is continuing to work on the mitigation for this issue. We are validating a potential fix.
Our next update will be within 2 hours.
December 5, 2022 10:43PM UTC
[Identified] A fix to mitigate new Automatic Speech Recognition (ASR) requests has been prepared and deployed to the European cloud as of Dec 5th 11:11 PM Central European time. Deployments to North America and Canada clouds will take place shortly after 5pm Pacific time today. Deployments for the Asia-Pacific and Australia clouds will take place at the end of business hours on Dec 6th.
We are working separately on a mitigation for ASR requests that failed prior to this fix.
Our next update will be within 3 hours
December 6, 2022 1:54AM UTC
[Identified] A fix has been prepared and a deployment is in progress for the North America and Canada clouds. We continue to work separately on a mitigation for ASR requests that failed prior to this fix.
Our next update will be within 3 hours.
December 6, 2022 2:53AM UTC
[Identified] A fix has been deployed to the North America and Canada clouds.
We continue to work separately on a mitigation for ASR requests that failed prior to this fix.
Our next update will be within 16 hours, at which point the fix will be deployed to the Asia-Pacific and Australia clouds.
