Summary
CI pipelines using git connectors with a Delegate as the mode of connection did not update the status back to SCM/Git providers. This did not impact those pipelines whose git connectors are connected via the Harness Platform.
Mitigation
Steps to resolve the issue immediately
- Rolled back delegate for rings to which customers belong.
- Rolled back delegate for all rings.
Detailed Timeline(PST
Time
Event
10/05/2023
3:30 AM
Delegate deployment
10:53 AM
First customer ticket for status check not reporting back
11-12 AM
Two more customer ticket for the same issue
12:20 PM
Incident channel created
2:20 PM
Reproduced the issue PR checks reporting not working via delegate
2:40 PM
Rolled back delegate in rings
3:30 PM
Informed customer about rollback
3:40PM
Customer confirmed restoration
RCA
Why didn't certain CI pipelines update SCM/Git status?
- The delegate task created to update the status on Git failed causing the status to be not reflected on Git provider.
Why did the delegate task fail?
- A change made to support the new Harness Code module introduced a missing dependency that failed during run-time.
Why didn't it impact all the CI pipelines?
- Harness provides two ways to connect to Git providers - via Harness Platform or via Harness Delegate. Only the CI pipelines using git connectors with a Delegate as the mode of connection did not update the status back to SCM/Git providers. This did not impact those pipelines whose git connectors are connected via the Harness Platform. This is because the missing dependency issue was in the Delegate.
Why was the missing dependency not predicted in the testing phase?
- We have automated tests that check running pipelines via Harness Delegate but the tests to check Git status updates in this mode of connectivity were missing.
Steps taken
Harness CI engineers tried to reproduce in-house with various combinations of infrastructure such as Kubernetes, Harness Cloud, Virtual Machines, etc but it took some time to realize this happens only when the git connector is set up to be connected via Harness Delegate instead of Harness Platform. As soon as we realized this, we engaged the delegate engineering team and they helped with reverting the delegate version to a previous one that did not have this code.
Follow-up actions
Add automation to catch this case and also set up internal alerts when the issue happens so things can be handled proactively.