Deployment Degradation - Slowness in Prod3

Incident Report for Harness

Resolved

This incident has been resolved.

Posted May 07, 2026 - 05:54 PDT

Update

We are continuing to monitor for any further issues.

Posted May 07, 2026 - 05:12 PDT

Update

A fix has been implemented and we are monitoring the system.

Posted May 07, 2026 - 05:12 PDT

Monitoring

A fix has been implemented and we are monitoring the results.

Posted May 07, 2026 - 05:09 PDT

Update

Executions are running fine now. However, there is still some slowness in the execution graph generation so we might see some delay in the execution updates to show up in UI.

But executions would work without any delay/slowness.

Posted May 07, 2026 - 05:06 PDT

Identified

We have identified the root cause as elevated queue depth and throughput saturation on the MongoDB layer, leading to processing latency. A fix is being implemented to optimize handling and restore database stability

Posted May 07, 2026 - 04:20 PDT

Update

We are continuing to investigate this issue.

Posted May 07, 2026 - 03:19 PDT

Investigating

We are currently investigating this issue.

Posted May 07, 2026 - 01:00 PDT

This incident affected: Prod 3 (Continuous Delivery (CD) - FirstGen - EOS, Continuous Delivery - Next Generation (CDNG), Cloud Cost Management (CCM), Continuous Error Tracking (CET), Continuous Integration Enterprise(CIE) - Self Hosted Runners, Continuous Integration Enterprise(CIE) - Mac Cloud Builds, Continuous Integration Enterprise(CIE) - Windows Cloud Builds, Continuous Integration Enterprise(CIE) - Linux Cloud Builds, Custom Dashboards, Feature Flags (FF), Security Testing Orchestration (STO), Service Reliability Management (SRM), Chaos Engineering, Internal Developer Portal (IDP), Infrastructure as Code Management (IaCM), Software Supply Chain Assurance (SSCA), Software Engineering Insights (SEI), Code Repository, Artifact Registry, Platform, FME).