Summary

On February 17, 2026, we had a traffic spike in one of the services in Prod3, which impacted the Pipeline Service’s capacity. We remediated this by addressing the source of the spike in workload and performing tuning of our backend systems.

Root Cause

Starting around 7:25 A.M. PST, our databases became overwhelmed with an increased rate of writes, causing resource pressure. The write latency spiked, causing our upstream systems to experience timeouts and errors.

Customer Impact

During the window of the impact

• Pipeline executions ran significantly slower or stalled, with initialization steps delayed.
• Slowness while performing CRUD operations on pipelines.

Resolution

We identified and disabled a high-frequency batch write workload that was contributing significantly to the write pressure. By switching that component to a lower-write alternative flow, full system recovery was confirmed at ~10:05 AM PST.

Prevention and Improvements

To prevent recurrence and enable faster identification of such issues, we are taking several measures:

• Automate the audit and proactively optimize resource-intensive queries. Optimize with better indexes or query scope limits to prevent working set overflow.
• Fine-tune workloads to increase headroom to handle spikes.
• Add proactive alerts for sustained traffic rates and resource utilization approaching the high watermark.
• Add capacity to our backend systems.