On October 6, 2026, pipelines containing an IaCM stage were unable to start on Prod1 for approximately two hours. Affected users saw an error when the pipeline attempted to create its execution plan. Pipelines that did not use an IaCM stage continued to run normally throughout.
The underlying cause was an internal logging defect in the service that handles IaCM stage execution: under a specific, rare timing condition, that service's own logging system could end up waiting on itself indefinitely, which in turn stalled the broader process responsible for preparing pipeline execution plans. The issue was triggered by an unusually large burst of log activity from one customer's pipeline. We resolved the immediate impact by rolling the service back to its previous stable version, and a permanent code fix is currently in testing.
The service responsible for IaCM pipeline stages logs its activity through an internal queue: log messages are placed on the queue, and a single background worker continuously reads from it and writes the messages out. Separately, that service has a filter that can rewrite the severity of certain log messages back onto that same queue.
Under a specific, narrow timing condition, this created a deadlock, if the queue happened to be completely full at the exact moment a rewritten message was being submitted back onto it, the one worker responsible for freeing up space on the queue would itself get stuck waiting for space to free up, a condition it could never resolve, since it was the only thing that could have freed that space. Once that happened, every other part of the service that needed to log anything also became stuck waiting, which cascaded into the pipeline-plan-creation process failing for any pipeline using an IaCM stage.