Overview :
CI Pipeline getting failed post rolling out new version -1.15.3
What was issue:
There was a change included in CI manager 1.15.3 build related to a code rewrite. Due to this change we had a scenario where there was backward incompatibility between two builds. Pipelines where plan creation happened on earlier version and execution on newer, failed.
Timeline:
| Time | Event |
|---|---|
| Feb 27 2024 3:09:33 PM IST | CI Manager new version was deployed to prod |
| Feb 27 2024 3:12 PM IST | CI Internal Sanity got failed, Internal incident created. |
| Feb 27 2024 3:22 PM IST | CI Manager reverted to older version |
Resolution:
We rolled back the release immediately as Internal Sanity failed.
RCA & Action Items:
Adding automated/manual check for deployment transition pipeline or execution such that we catch these issues ahead of time.