CD Pipeline execution slowness in Prod3 cluster

Incident Report for Harness

Postmortem

Summary

On December 22, 2025, between 10:55 AM and 2:00 PM CET, customers experienced slowness when accessing and editing pipelines in the Prod3 environment, particularly for pipelines using Harness Code repositories.

Root Cause

The issue was caused by a resource-intensive backend operation that led to elevated system load, resulting in degraded performance for affected workflows.

Impact

Customer impact: Slowness when accessing and editing pipelines
Duration: December 22, 2025, from 10:55 AM CET to 2:00 PM CET
Scope: CODE module and related usage metrics
No data loss or pipeline execution failures were observed.

Mitigation

The resource-intensive feature was disabled, which immediately restored normal system performance.

Next Steps

Permanent fix: Optimize backend systems to prevent similar resource contention issues.
Monitoring: Add enhanced monitoring and alerting to detect early signs of resource saturation.
Testing improvements: Strengthen pre-production testing to identify and prevent potential resource leaks before deployment.

Posted Jan 05, 2026 - 09:59 PST

Resolved

We have implemented a fix which resolved the issue. We will be providing postmortem report on this issue.

Posted Dec 22, 2025 - 05:02 PST

Update

We are continuing to investigate this issue.

Posted Dec 22, 2025 - 03:25 PST

Investigating

We are currently investigating slow pipeline executions in the Prod-3 cluster. The issue appears to affect only pipelines that are stored in Harness Code repositories.

Posted Dec 22, 2025 - 02:38 PST

This incident affected: Prod 3 (Continuous Delivery - Next Generation (CDNG)).