Overview:
Hosted CI Builds on MacOS are failing to initialise in all environments.

What was the issue?

• To fix the registry issue which was recurring on 10 Feb 2024. https://manage.statuspage.io/pages/zmnt6tkys0q0/incidents/sd82v3lmcqqd#postmortem  we attempted a release for Anka controller.
• Controller would be required to initialise the VM’s and manage the state of the VM’s, it being down means the Mac pipelines were not functional.
• Post debugging we identified that there were network configurations issue that had to be re-configured to ensure the controller was accessible.

Timeline:

Time Event 22nd Feb 2024 8:13 AM IST We started deployment for Controller to fix issue 22nd Feb 2024 8:24 AM IST We noticed Controller was not coming up, Hence started revert of release. 22nd Feb 2024 8:30 AM IST Revert did not work here, hence Internally incident was created and an investigation started. 22nd Feb 2024 12:11 PM IST Dlite deployment - bounce done with new network changes on all cluster.

Resolution:

We resolved network configurations related issues for controller, further it was accessible.

RCA & Action Items:

As part of the improvements, we will be moving this to a high availability setup. We will also be updating the alerting and monitoring around this workflow to capture such issues immediately.