Overview:
Hosted CI Builds on MacOS are failing to initialise in all environments.

What was the issue?

• To fix the registry issue which was recurring on 10 Feb 2024. https://manage.statuspage.io/pages/zmnt6tkys0q0/incidents/sd82v3lmcqqd#postmortem  we attempted a release for Anka controller.
• Controller would be required to initialise the VM’s and manage the state of the VM’s, it being down means the Mac pipelines were not functional.
• Post debugging we identified that there were network configurations issue that had to be re-configured to ensure the controller was accessible.

Timeline:

Resolution:

We resolved network configurations related issues for controller, further it was accessible.

RCA & Action Items:

As part of the improvements, we will be moving this to a high availability setup. We will also be updating the alerting and monitoring around this workflow to capture such issues immediately.