CI pipeline are getting failed with latest release

Incident Report for Harness

Postmortem

Overview :
CI Pipeline getting failed post rolling out new version -1.15.3

What was issue:
There was a change included in CI manager 1.15.3 build related to a code rewrite. Due to this change we had a scenario where there was backward incompatibility between two builds. Pipelines where plan creation happened on earlier version and execution on newer, failed.

‌

Timeline:

Time Event Feb 27 2024 3:09:33 PM IST CI Manager new version was deployed to prod Feb 27 2024 3:12 PM IST CI Internal Sanity got failed, Internal incident created. Feb 27 2024 3:22 PM IST CI Manager reverted to older version

Resolution:
We rolled back the release immediately as Internal Sanity failed.

RCA & Action Items:

Adding automated/manual check for deployment transition pipeline or execution such that we catch these issues ahead of time.

Posted Feb 28, 2024 - 06:41 PST

Resolved

Issue was with latest release, reverted to old release.

Posted Feb 27, 2024 - 02:18 PST

Investigating

We are currently investigating this issue.

Posted Feb 27, 2024 - 02:17 PST

This incident affected: Prod 2 (Continuous Integration Enterprise(CIE) - Self Hosted Runners, Continuous Integration Enterprise(CIE) - Mac Cloud Builds).