CI Expressions(nested expressions) are impacted.

Incident Report for Harness

Postmortem

Summary

Some customer pipelines failed to initialize when using certain stage-level expressions. The issue was limited to specific pipelines containing a particular nested expression pattern. Only a small number of customers reported failures, and other pipelines continued to run normally.

Root Cause

A recent update changed how certain expressions were evaluated during pipeline initialization. In pipelines that used nested expressions dependent on runtime values, those values were not yet available at initialization time. This caused the expression evaluation to fail, resulting in the pipeline failing to start.

Impact

A subset of customer pipelines containing specific nested expressions were unable to initialize for several hours. Pipelines without these expressions continued to run successfully. No data loss occurred, and no other product areas were affected.

Remediation

Immediate

Rolled back to the last known-good release, restoring pipeline functionality for affected customers.

Permanent

Deployed an updated build with improved expression-handling logic to ensure that expressions relying on runtime data are only evaluated when those values are available.

Action Items

  • Add safeguards to prevent early evaluation of expressions requiring runtime values.
  • Expand automated test coverage to include additional expression patterns used by customers.
  • Strengthen validation to ensure future updates do not introduce similar expression-handling regressions.
Posted Dec 09, 2025 - 11:06 PST

Resolved

This incident has been resolved.
Posted Dec 05, 2025 - 08:33 PST

Update

We are continuing to monitor for any further issues.
Posted Dec 05, 2025 - 08:26 PST

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Dec 05, 2025 - 08:16 PST

Update

We are continuing to work on a fix for this issue.
Posted Dec 05, 2025 - 08:02 PST

Identified

The issue has been identified and a fix is being implemented.
Posted Dec 05, 2025 - 04:42 PST

Update

We are continuing to investigate this issue.
Posted Dec 05, 2025 - 03:07 PST

Investigating

We are currently investigating this issue.
Posted Dec 05, 2025 - 00:56 PST
This incident affected: Prod 3 (Continuous Integration Enterprise(CIE) - Self Hosted Runners, Continuous Integration Enterprise(CIE) - Mac Cloud Builds, Continuous Integration Enterprise(CIE) - Windows Cloud Builds, Continuous Integration Enterprise(CIE) - Linux Cloud Builds), Prod 2 (Continuous Integration Enterprise(CIE) - Self Hosted Runners, Continuous Integration Enterprise(CIE) - Mac Cloud Builds, Continuous Integration Enterprise(CIE) - Windows Cloud Builds, Continuous Integration Enterprise(CIE) - Linux Cloud Builds), and Prod 1 (Continuous Integration Enterprise(CIE) - Self Hosted Runners, Continuous Integration Enterprise(CIE) - Mac Cloud Builds, Continuous Integration Enterprise(CIE) - Windows Cloud Builds, Continuous Integration Enterprise(CIE) - Linux Cloud Builds).