Overview

We received a report from one of our customers about issues with their pipeline executions using step groups in our Prod-1 cluster. Details about the incident are below.

Timeline (PST)

Time Description 11:52 am PST One customer reported an issue with their pipeline executions involving step groups. 12:11 pm PST The issue was identified with the previous deployment. 12:15 pm PST Rollback was initiated for the impacted module. 12:20 pm PST Rollback completed. 12:30 pm PST The customer confirmed on issue resolution.

Resolution

Harness engineers performed a rollback of the latest deployment for one of the internal modules, which immediately eliminated the failure with Expression Engine and resolved the issue with pipeline executions.

RCA

We have identified the problem with our Expression Engine flow where a crucial condition accounting for the possibility of metadata being null if a step, step group, or stage did not have a strategy defined. The issue happens when a specific setting ‘Enable Json Support for expressions’ is enabled.

We have resolved the incident by performing the rollback of the latest deployment for the affected module which happened earlier in the day.

Action Items

1. Our automation suite will be enhanced to capture scenarios for the Expression Engine flow and also include the expressions used by our customers.
2. Update our documentation on the best practices for utilising expressions.