Pipelines and dashboards are impacted in Prod2

Summary

On March 11, 2026, customers experienced pipeline failures and degraded UI performance(incorrect status of states) and CCM Dashboards were not accessible to the affected customers in the Prod2 environment. The issue was caused by a degradation in an internal shared infrastructure component used for coordination across services.

The incident began around 7:10 AM PST and was fully mitigated by approximately 10:12 AM PST. During this period, pipeline execution throughput was significantly impacted for affected customers.

Root Cause

The issue was caused by resource saturation in a shared infrastructure component used for distributed coordination, which led to increased latency and failures in service-to-service communication.

As a result, pipeline execution services were unable to process workloads efficiently, leading to a buildup of queued tasks and reduced system throughput.

Impact

Customers experienced the following:

• Pipeline executions failing or not progressing
• Increased pipeline execution times
• UI delays due to processing backlogs

The impact was limited to specific production environments and no data loss occurred.

Mitigation

Immediate

• Redirected services to a higher-capacity infrastructure instance to restore normal processing
• Cleared accumulated processing backlogs to recover system throughput
• Scaled supporting services to stabilize performance

Permanent

• Improved monitoring and alerting for early detection of resource saturation
• Implemented capacity and scaling improvements to handle higher load scenarios
• Initiated architectural improvements to reduce reliance on shared coordination components

Action Items

To prevent such issues from happening again we are taking several steps:

• Enhance alerting to detect early signs of infrastructure saturation
• Review and optimize system behavior under high concurrency scenarios
• Continue investigation into the triggering conditions and incorporate findings into long-term improvements