Overview

Mac Builds started failing during initialize step due to a timeout on trying to connect to the mac cloud provider.

‌

Timeline (PST)

Time Event 12:08 PM IST OCT 16 2023 Engineering got message from a customer about Mac Builds failing via Zendesk. 12:35 PM IST OCT 16 2023 Internal FireHydrant Incident created to check the failed pipelines for customers. 12:40 PM IST OCT 16 2023 Bounced dlite deployment with config changes to remove mac pool in prod2 environment.

‌

Resolution

Bounced dlite deployment with config changes to remove mac pool in the impacted environment.

‌

Affected Users

This impacted the customers in the prod2 environment using the hosted mac-builds.

‌

RCA

As part of a prior incident we added a set of additional NAT IP’s. These NAT IP’s were not whitelisted in another project which is used to initialize the MAC VMs. The whitelisting was only required for the project which is used to initialize MAC builds because we use a different cloud provider for MAC when compared to linux/windows.