Impact:
Harness custom dashboards were intermittently inaccessible during the time of the incident.
Root Cause:
Custom Dashboards microservice in the harness cluster of services had experienced a server overloaded with requests causing a CPU spike resulting in a crash and a restart and due to the high volume of requests the UI was inaccessible.
Timeline:
Time
Event
May 16 at 12:32 PM PST
Internal team noticed slowness in custom dashboards
May 16 at 12:32 PM PST
Internal call was initiated and engineering started triaging
May 16 at 12:41 PM PST
Functionality returned back to normal after a pod restart
Action items:
Adopting the Gunicorn server to handle more requests per second and added resiliency
Posted May 18, 2023 - 20:51 PDT
Resolved
We can confirm normal operation. Get Ship Done!
We will continue to continue to monitor and ensure stability.
Posted May 16, 2023 - 13:15 PDT
Monitoring
Harness service issues have been addressed and normal operations have been resumed. We are monitoring the service to ensure normal performance continues.
Posted May 16, 2023 - 12:41 PDT
Investigating
Harness NG Dashboards in Prod 2 cluster are unable to load. We are working to identify the cause and restore normal operations as soon as possible.