Hosted CI Builds Failing Intermittently

Summary

Between May 27 and June 1, 2026, some Harness CI customers experienced pipeline execution failures with the error:

failed to call LE.RetryStartStep: context deadline exceeded

Impact

Affected customers saw intermittent CI pipeline failures during step execution.

Existing pipeline definitions, customer data, source code, and artifacts were not impacted.

Root Cause

The root cause was a deadlock in the Light Engine logging path. When the log service returned an error, the Light Engine log writer attempted to reacquire a mutex it already held. This caused the Light Engine process to freeze, which led to step execution timeouts and pipeline failures.  

Mitigation and Resolution

Harness Engineering took multiple mitigation steps during the incident, including:

• Rolled back affected runner versions where needed
• Increased relevant timeout configurations
• Reduced log-service load and latency
• Temporarily disabled the affected livelog streaming path
• Migrated selected workloads across regions and infrastructure providers
• Pinned a fixed Light Engine version through runner configuration

The final fix addressed the mutex deadlock in the Light Engine log writer and prevented the same lock from being reacquired while already held.  

Prevention and Follow-Up Actions

Harness is taking the following actions to reduce recurrence risk:

• Improve deadlock detection in critical concurrent code paths
• Strengthen error handling for log-service interactions
• Add better monitoring for Light Engine process health
• Improve safeguards around logging-path failures
• Continue reviewing runner rollout and validation processes