An internal monitoring script overload caused our build scheduler to run out of memory, briefly failing few Mac pipelines; the script’s been removed and safeguards are in progress.
A watcher script deployed to Mac clients ran commands every few minutes that flooded the server’s memory and caused it to go OOM. This lead to pipeline execution failures for Mac Hosted builds. The script was removed, deployment pipelines were fixed, and a log-based alternative plus additional safeguards/alerts are being implemented.
We immediately removed the watcher script issuing the heavy client calls, which stopped the memory surge and stabilized the server. Following the scheduler restart, the Mac Pipelines recovered and system was functional.