Summary

On April 8, 2026, customers in Prod1 and Prod2 experienced degraded performance when logging into the Harness platform. Additionally, in Prod2, customers were unable to start new pipeline executions and some running pipelines failed. The issue lasted approximately 1 hour and 35 minutes. 

Root Cause

The issue was caused by a sudden surge of task reassignment requests triggered after customer delegate restarts. This resulted in a high volume of backend processing requests that exceeded expected limits, leading to elevated resource utilization and degraded performance of the Harness Manager service. 

Impact

• Customers in Prod1 and Prod2 experienced login failures and degraded user operations.
• Customers in Prod2 were unable to start new pipeline executions, and some ongoing executions failed.
• All customers in the affected clusters experienced service degradation during the incident window. 

Remediation

Immediate:

• Restarted affected services and stabilized system performance, restoring login and pipeline functionality.

Permanent:

• Introduced safeguards to limit backend processing for large task reassignment scenarios.
• Identifying and applying limits to similar high-volume operations to prevent resource exhaustion. 

Action Items

To prevent from such issues from happening again

• Implement query limits for high-volume task processing scenarios.
• Audit and enforce limits across similar backend operations so that we can be resilient.
• Enhance monitoring and alerting for abnormal spikes in task reassignment and resource utilization.