All Systems Operational

Website




Operational

Video playback




Operational

Search




Operational

Recording / Uploading




Operational

Processing / Encoding




Operational

Integrations




Operational

External Services

AWS CloudFront

AWS EC2

AWS Route 53

AWS S3

Amazon Web Services

SendGrid

WebEx

Zoom

Locations

History (Last 7 days)

Panopto EU Cloud Scheduled DowntimePlanned Maintenance

Description

On Tuesday, August 9th, your Panopto Cloud site will be taken offline for database maintenance. This upgrade will occur from 10:00 pm to 10:10 pm GMT. We expect up to 10 minutes of downtime. Best Regards, Your Team at Panopto


Components

Website, Video playback, Search, Recording / Uploading, Processing / Encoding, Integrations


Locations

Europe (EU)


Schedule

August 9, 2022 10:00PM - August 9, 2022 10:10PM UTC


August 9, 2022 10:00PM UTC
[Update] On Tuesday, August 9th, your Panopto Cloud site will be taken offline for database maintenance. This upgrade will occur from 10:00 pm to 10:10 pm GMT. We expect up to 10 minutes of downtime. Best Regards, Your Team at Panopto

August 9, 2022 10:18PM UTC
[Update] We have completed the maintenance to the EU Cloud. Best Regards, Your Panopto Team
Panopto NA Cloud Scheduled DowntimePlanned Maintenance

Description

On Tuesday, August 9th, your Panopto Cloud site will be taken offline for database maintenance. This upgrade will occur from 1:00 am to 1:10 am EDT. We expect up to 10 minutes of downtime. Best Regards, Your Team at Panopto


Components

Website, Video playback, Search, Recording / Uploading, Processing / Encoding, Integrations


Locations

North America (NA)


Schedule

August 9, 2022 5:00AM - August 9, 2022 5:10AM UTC


August 9, 2022 5:00AM UTC
[Update] On Tuesday, August 9th, your Panopto Cloud site will be taken offline for database maintenance. This upgrade will occur from 1:00 am to 1:10 am EDT. We expect up to 10 minutes of downtime. Best Regards, Your Team at Panopto

August 9, 2022 5:31AM UTC
[Update] We have completed the maintenance to the NA Cloud. Best Regards, Your Panopto Team
Panopto NA Cloud Notification 2022-07-31Service Disruption

Incident Status

Service Disruption


Components

Search


Locations

North America (NA)



August 1, 2022 3:32AM UTC
[Investigating] The Panopto NA Cloud is experiencing an increased error rate in search. We’re sorry for this interruption. Our cloud operations team is urgently investigating the issue and we’ll update here as soon as we know more. Our next update will be within 30 minutes. What does this mean? Searching for content may fail to succeed.

August 1, 2022 4:04AM UTC
[Monitoring] The source of the increased error rates has been mitigated. Our cloud operations team is monitoring cloud metrics to ensure the issue is resolved. Our next update will be within 60 minutes.

August 1, 2022 4:55AM UTC
[Resolved] The Panopto NA Cloud has been stable and metrics have remained normal since our last update. We are resolving this incident and Panopto engineers are completing a root cause analysis, which we will post here, within the next week.

August 4, 2022 7:09PM UTC
[Resolved] On 2022-08-01 from 3:20 to 3:46 UTC, the Panopto NA Cloud experienced increased error rates in search. During this time, customers will have experienced errors when searching for content. RCA: Due to an issue in our search service, some search servers began to slowly consume excessive resources over time. Eventually, the servers became overloaded and began failing to serve requests. Upon being alerted to this, we resolved the issue by fully resetting the impacted servers. To prevent a recurrence of this issue, we will fix this excessive resource consumption issue in our search service. Additionally, we will add additional alarms to detect such excessive resource consumption before it causes issues in the future.
Panopto EU Cloud Notification 2022-07-27Service Disruption

Incident Status

Service Disruption


Components

Search


Locations

Europe (EU)



July 28, 2022 3:28AM UTC
[Investigating] The Panopto EU Cloud is experiencing an increased error rate in search. We’re sorry for this interruption. Our cloud operations team is urgently investigating the issue and we’ll update here as soon as we know more. Our next update will be within 30 minutes. What does this mean? Search queries will result in timeouts or errors.

July 28, 2022 4:20AM UTC
[Resolved] The Panopto EU Cloud experienced increased error rates in search on 2022-07-28, from 03:07 to 03:34 UTC. We have resolved this incident and Panopto engineers are completing a root cause analysis, which we will post here, within the next week.

August 4, 2022 7:08PM UTC
[Resolved] On 2022-07-27 from 15:03 to 15:25 UTC, and on 2022-07-28 from 3:07 to 3:34 UTC, the Panopto EU Cloud experienced increased error rates in search. During this time, customers will have experienced errors when searching for content. RCA: Due to an issue in our search service, some search servers began to slowly consume excessive resources over time. Eventually, the servers became overloaded and began failing to serve requests. Upon being alerted to this, we mitigated the issue by resetting the impacted servers. Later, we used a maintenance tool to attempt to fully repair the service. This maintenance tool had a bug in it which triggered the second outage. We resolved that second outage by restarting the impacted servers. To prevent a recurrence of this issue, we will fix this excessive resource consumption issue in our search service. Additionally, we will fix the bug in our maintenance tool that caused the second outage. Finally, we will add additional alarms to detect such excessive resource consumption before it causes issues in the future.
Panopto EU Cloud Notification 2022-07-27Operational

Incident Status

Operational


Components

Search


Locations

Europe (EU)



July 27, 2022 5:00PM UTC
[Resolved] The Panopto EU Cloud experienced increased error rates in search on 2022-07-27, from 15:03 to 15:25 UTC. During this time, users will have experienced errors when searching for content. We have resolved this incident and Panopto engineers are completing a root cause analysis, which we will post here, within the next week.

August 4, 2022 7:07PM UTC
[Resolved] On 2022-07-27 from 15:03 to 15:25 UTC, and on 2022-07-28 from 3:07 to 3:34 UTC, the Panopto EU Cloud experienced increased error rates in search. During this time, customers will have experienced errors when searching for content. RCA: Due to an issue in our search service, some search servers began to slowly consume excessive resources over time. Eventually, the servers became overloaded and began failing to serve requests. Upon being alerted to this, we mitigated the issue by resetting the impacted servers. Later, we used a maintenance tool to attempt to fully repair the service. This maintenance tool had a bug in it which triggered the second outage. We resolved that second outage by restarting the impacted servers. To prevent a recurrence of this issue, we will fix this excessive resource consumption issue in our search service. Additionally, we will fix the bug in our maintenance tool that caused the second outage. Finally, we will add additional alarms to detect such excessive resource consumption before it causes issues in the future.