Delay is observed while loading the Dashboards in Prod2 cluster

Incident Report for Harness

Postmortem

Overview

Some widgets on our service dashboards were slow to load or timing out, and custom dashboards were also taking longer to load. The impact was limited to prod 2 customers only.

Timeline (PST)

Time Event 3:20 AM Incident declared and team got engaged. 4.00 AM Issue got identified and working on mitigation. 5.00 AM Mitigated and monitoring. 5:30 AM Issue got resolved

RCA

Certain database queries were taking significant amount of time to complete, leading to a spike in CPU usage, with levels reaching nearly 100 percent. To address this issue, the following actions were taken as part of the resolution process:

We terminated long-running queries and optimized database to address the issue.
We added database CPU alerts in addition to our existing APM alerts to further improve our MTTD.

Follow up Action Items

Set up a cron job to perform database optimization on a regular basis.
Review and optimize other queries.

Posted Oct 27, 2023 - 10:45 PDT

Resolved

We can confirm normal operation. Get Ship Done!
We will continue to monitor and ensure stability.

Posted Aug 17, 2023 - 05:01 PDT

Monitoring

Harness service issues have been addressed and normal operations have been resumed. We are monitoring the service to ensure normal performance continues.

Posted Aug 17, 2023 - 04:11 PDT

Update

We are continuing to work on a fix for this issue.

Posted Aug 17, 2023 - 04:07 PDT

Identified

We have identified a potential cause of the service issues and are working hard to address it. Please continue to monitor this page for updates.

Posted Aug 17, 2023 - 03:59 PDT

Investigating

Delay is observed while loading the Dashboards in Prod2 cluster. We are working to identify the cause and restore normal operations as soon as possible.

Posted Aug 17, 2023 - 03:24 PDT

This incident affected: Prod 2 (Custom Dashboards).