Hosted CI Builds on MacOS are failing to initialize in all environments

Incident Report for Harness

Postmortem

Overview

Hosted CI Builds on MacOS are failing to initialize in all environments.

Timeline

Resolution

A server used for MacOs build farm orchestration caused the image repository to be unavailable. The server was made operational and the system restored.

RCA & Action Items

As part of the improvements, we will be moving this to a high availability setup. We will also be updating the alerting and monitoring around this workflow to capture such issues immediately.

Posted Feb 12, 2024 - 11:01 PST

Resolved

Systems are back to operational. Postmortem will contain details and identified action items.

Posted Feb 11, 2024 - 19:35 PST

Investigating

We are currently investigating this issue.

Posted Feb 11, 2024 - 19:20 PST

This incident affected: Prod 3 (Continuous Integration Enterprise(CIE) - Mac Cloud Builds), Prod 1 (Continuous Integration Enterprise(CIE) - Mac Cloud Builds), and Prod 2 (Continuous Integration Enterprise(CIE) - Mac Cloud Builds).