Summary

On March 11, 2026, customers experienced pipeline failures and degraded UI performance(incorrect status of states) in the Prod2 environment. The issue was caused by a degradation in an internal shared infrastructure component used for coordination across services.

The incident began around 7:10 AM PST and was fully mitigated by approximately 10:12 AM PST. During this period, pipeline execution throughput was significantly impacted for affected customers.

Root Cause

The issue was caused by resource saturation in a shared infrastructure component used for distributed coordination, which led to increased latency and failures in service-to-service communication.

As a result, pipeline execution services were unable to process workloads efficiently, leading to a buildup of queued tasks and reduced system throughput.

Impact

Customers experienced the following:

• Pipeline executions failing or not progressing
• Increased pipeline execution times
• UI delays due to processing backlogs

The impact was limited to specific production environments and no data loss occurred.

Mitigation

Immediate

• Redirected services to a higher-capacity infrastructure instance to restore normal processing
• Cleared accumulated processing backlogs to recover system throughput
• Scaled supporting services to stabilize performance

Permanent

• Improved monitoring and alerting for early detection of resource saturation
• Implemented capacity and scaling improvements to handle higher load scenarios
• Initiated architectural improvements to reduce reliance on shared coordination components

Action Items

To prevent such issues from happening again we are taking several steps:

• Enhance alerting to detect early signs of infrastructure saturation
• Review and optimize system behavior under high concurrency scenarios
• Continue investigation into the triggering conditions and incorporate findings into long-term improvements