All Systems Operational

Website




Operational

Video playback




Operational

Search




Operational

Recording / Uploading




Operational

Processing / Encoding




Operational

External Services

AWS EC2

AWS S3

Locations

History (Last 7 days)

Incident Status

Operational


Components

Processing / Encoding


Locations

EU (Ireland)




April 24, 2018 8:46AM UTC
[Investigating] We're investigating increased error rates on the status of processing jobs in the Panopto Cloud. This is currently impacting some queued jobs in the processing area on your Panopto cloud site. This will not impact current recordings processing and becoming available. The issue is only visible to administrators and all sessions in the 'queued' state should still be able to be viewed. Panopto engineers are investigating the issue, and we will update in due course. This issue should not have any detrimental effects to end users.

April 24, 2018 10:14PM UTC
[Resolved] Panopto issued a bug alert around on 2018-02-26 UTC regarding an issue caused when Chrome v64's media playback engine changed. As part of that change, playback would halt when the browser encountered minor inconsistencies in the audio stream. This affected a non-trivial subset of Panopto sessions created by Panopto for Mac recorders. On 2018-04-20 UTC we launched an initial cleanup of affected content on the Panopto EU Cloud. This first cleanup pass resulted in reporting of failures on a substantial subset of content, leading to an error message being visible through the web view but otherwise causing no viewing or content access issues. At 2018-04-23 19:00 UTC we re-initiated the cleanup process and cleared the error messages created during the first iteration. This resulted in "Processing" messages being displayed to session creators, videographers, and administrators. The jobs associated with the latest round of cleanup will finish processing by 2018-04-29 03:00 UTC. Upon processing completion, the associated session messages will disappear. The Panopto Operations team will monitor the failure rate and ensure any content that fails this processing will be handled and cleaned up. Throughout this process all user interactions with the content will be unaffected, including viewing. We will work to improve our system so that confusing messages like this will not be displayed in the future.