Overview

A small subset of pipelines encountered VM initialization failures on Cloud Mac builds.

Timeline (PST)

Time Event 1:17 AM Customer reported this issue via Support ticket 1:29 AM Internal incident created 1:32 AM Identified that we hit full capacity in Mac pools 2:11 AM We completed a manual cleanup of stale VM’s before the scheduled cleanup, making sure customer pipelines were able to proceed 2:30 AM Mac pool was returned to normal state

‌

Resolution

Manual cleanup of stale VM’s before the scheduled cleanup to allow the active pipelines to complete.

Affected accounts

One of the customers using Mac hosted builds in Prod2 was impacted.

RCA

During the VM provisioning process, the VMs were not cleaned up as they should have been when an abort call was made. This issue arose from a recent modification we made to address other cleanup issues. The situation would have been managed effectively if appropriate error handling and cleanup had been incorporated into the Anka driver (Driver for mac OS deployments) to VM creation logic. However, as the Anka driver returned an error, we didn't create a database entry for cleanup, which in turn led to the dlite instance not performing the necessary cleanup due to the absence of a database entry.

Action Items

• Anka create API needs to be made Idempotent
• Additional monitoring/alerting for stale VMs