Pipeline Updates in Prod4 is taking time.

On May 14, 2026 , some customers running pipelines in the Prod4 production environment observed pipeline create and update requests that were slow or failed, and pipeline stages that did not start, produced no logs, and were eventually auto-aborted as “stuck”.

The issue was caused by an underlying compute node in our Prod4 cluster being recycled abruptly.

Impact

During the incident window (approximately 5:38 PM PDT on May 14 to 9:47 PM PDT on May 14, 2026):

• Some pipeline create and update requests on Prod4 were slow or failed.
• Some Prod4 pipeline executions hung at the stage-start step, producing no logs, and were eventually auto-aborted as “stuck” after a timeout.
• Behavior was intermittent — only pipelines whose requests were routed to an affected service pod were impacted; other pipelines continued to execute normally.

There was no data loss. The majority of pipelines on Prod4 continued to execute successfully throughout the incident — the primary impact was that affected create/update requests slowed down or failed, and a subset of pipelines could not progress and had to be aborted and re-run after mitigation. Overall service availability was degraded during this window.

Root Cause

During the incident, an underlying compute node in our Prod4 cluster was recycled by the cloud provider without completing its normal graceful-drain process, so the supporting-service pods running on that node were terminated abruptly. As a result, in-flight requests from the backend service to those pods were left without a response.

Mitigation

Harness completed the following immediate mitigation steps:

• Restarted the affected supporting-service pods to restore healthy targets.
• Restarted the pipeline service in Prod4 to clear the blocked worker threads. This is what fully restored normal pipeline create/update behavior and stage-start behavior; restarting only the supporting service was not enough on its own.
• Confirmed pipeline executions returned to normal and updated the status page to mitigated.

These actions restored pipeline execution behavior and resolved the customer-facing impact.

Action Items

To reduce the risk of recurrence and improve detection, the following actions are in various stages of being implemented:

• Optimize timeouts to the pipeline service’s plan-creation requests so that when a supporting service goes away unexpectedly, the worker threads recover automatically instead of remaining blocked.
• Investigate the abrupt node-recycle behavior in Prod4 with our cloud provider to ensure pods running on a recycled node receive a graceful shutdown signal in the future.
• Add proactive paging alerts on service worker-thread saturation, so this failure mode is detected before it becomes a impacting issue