Summary

On May 20 and May 21, 2026, customers on the Prod3 cluster experienced degraded pipeline performance on two separate occasions. In both cases, pipeline executions ran slower than usual and execution graph rendering was delayed because database operations were being queued on the backing MongoDB instance. Both incidents were mitigated by adjusting database write settings and adding capacity headroom, which relieved pressure on MongoDB and allowed normal operations to resume.

Root Cause

A higher-than-usual rate of concurrent updates against a frequently modified document used during pipeline execution produced a high rate of write conflicts in MongoDB. Each conflict triggered an automatic retry, which compounded under load and drove CPU utilization on the database to a sustained high level.

As CPU saturated:

• MongoDB started queuing incoming commands instead of executing them immediately.
• Operations dependent on those writes — pipeline step transitions and execution graph generation — slowed down or stalled while waiting for the queue to drain.

Customer Impact

Customers on Prod3 were impacted during two windows:

• May 20, 2026: ~3:15 AM to ~4:45 AM PST (~1h 30m).
• May 21, 2026: ~6:42 AM to ~7:47 AM PST (~1h 5m).

In both windows:

• Pipeline executions ran significantly slower, with delays in step initialization and execution graph generation.
• Pipelines with tight configured timeouts may have timed out as a result of the slowness.

No execution failures were observed beyond timeout-related effects, and no full outage occurred. All Harness modules that rely on Pipelines on Prod3 were affected for the duration of each incident.

Resolution

After identifying that database contention was the bottleneck, we adjusted the pipeline service's database write settings to reduce overhead on each operation and added additional database capacity to absorb the load. Together these reduced write conflicts and allowed the queued commands to drain, after which pipeline operations returned to normal.

Prevention and Improvements

1. Strengthen concurrency control around the high-contention update path so that concurrent updates to the same document no longer race in a way that produces write conflicts at scale.
2. Strengthen load and concurrency testing on hot execution paths so similar contention patterns are caught before they affect production traffic.
3. Improve proactive alerting on database write conflicts, queued commands, and sustained CPU utilization so the team can intervene before customer-facing latency is affected.