Pipeline executions are observing degraded performance

Summary

On Wednesday, September 9, 2025, between 8:58 AM and 9:49 AM PST, customers experienced slower pipeline runs in the Prod2 cluster. Pipelines that started during this time took longer than usual to finish. All pipelines eventually completed, and no other parts of the product or service were affected.

Root Cause

During this period, there was a sudden and a sharp increase in the number of pipeline runs. This extra load slowed down the database, which in turn made the backend services slower. As a result, customers saw delays in their pipeline executions.

Impact

• Pipelines that were started during the incident window ran slower than normal.
• Individual stages and steps also took longer to begin, and this delay was visible in the Harness UI.

‌

‌

Remediation

• Immediate: Scaled database server to handle increased load

‌

Action Items

1. Optimize Expression Engine to reduce database usage so that our Databases have more headroom and can handle spikes
2. Optimize the recovery time by identifying the issue early and do preventive scale ups