Partial Service Disruption
Website, Video playback, Search, Recording / Uploading, Processing / Encoding, Integrations
November 29, 2023 5:03PM UTC
[Resolved] From 12:30 pm UTC on 11/15 to 4:00 am UTC on 11/18, the Panopto NA Cloud experienced intermittent increased error rates impacting a subset of customers.
RCA: The higher errors rates were caused by issues connecting to one of the database servers. On the morning of 11/18, the database server was upgraded with additional CPU and memory capacity. Since the upgrade was performed there have been no further issues.
November 27, 2023 5:25PM UTC
[Monitoring] The system is operating normally. Our cloud operations team is continuing to monitor cloud metrics to ensure the issue is resolved.
Our final update will be at 9 am PST, Wednesday, 11/29.
November 22, 2023 11:12PM UTC
[Monitoring] The system is operating normally. Our cloud operations team is continuing to monitor cloud metrics to ensure the issue is resolved.
Our next update will be at 9 am PST, Monday, 11/27.
November 20, 2023 4:19PM UTC
[Monitoring] The system is operating normally. Our cloud operations team is continuing to monitor cloud metrics to ensure the issue is resolved.
Our next update will be at 5 pm PST, Wednesday, 11/22.
November 18, 2023 5:04PM UTC
[Monitoring] The database maintenance for North America (NA) has successfully completed.
The source of the increased error rates has been mitigated. Our cloud operations team is monitoring cloud metrics to ensure the issue is resolved.
Our next update will be within 48 hours.
November 18, 2023 3:01PM UTC
[Identified] The database maintenance for North America (NA) has started.
November 18, 2023 12:18AM UTC
[Identified] The source of the increased error rates has been identified. Our cloud operations team is working on a mitigation for the issue.
As part of applying this mitigation and resolving this issue, the NA Cloud will be unavailable between 7am PST and 9am PST on Saturday, November 18th, 2023.
Our next update will be when the maintenance begins at 7am PST on Saturday, November 18th, 2023.
November 17, 2023 8:23PM UTC
[Investigating] The Panopto NA Cloud is intermittently experiencing increased error rates.
We’re sorry for this interruption. Our cloud operations team is urgently investigating the issue and we’ll update here as soon as we know more.
Our next update will be within 4 hours.
What does this mean?
General website usage, Login, Video Playback, Search, Recording/Uploading, Processing/Encoding, and Integrations may be unstable and attempts to login or edit content may fail.
