An internal monitoring script overload caused our build scheduler to run out of memory, briefly failing few Mac pipelines; the script’s been removed and safeguards are in progress.
A watcher script deployed to Mac clients ran commands every few minutes that flooded the server’s memory and caused it to go OOM. This lead to pipeline execution failures for Mac Hosted builds. The script was removed, deployment pipelines were fixed, and a log-based alternative plus additional safeguards/alerts are being implemented.
| Timeline | Event |
|---|---|
| 2025-07-03 18:37 UTC | FH started to investigate the internal alerts and customer reported failures |
| 2025-07-03 18:53:50 UTC | Issue identified to be related to a monitoring script |
| 2025-07-03 19:49:30 UTC | Issue mitigated and monitoring script stopped |
We immediately removed the watcher script issuing the heavy client calls, which stopped the memory surge and stabilized the server. Following the scheduler restart, the Mac Pipelines recovered and system was functional.