CI/STO Stage Failures

Summary
On June 6th, our SCM service experienced degraded performance that briefly impacted infrastructure pipeline operations. This was triggered by a high-volume onboarding of modules using SSH-based connectors via our Terraform provider, which caused an unexpected resource spike in core services. This effected all services that use the SCM service including CI, CD & STO.

Root Cause
The primary cause was a surge in SSH metadata validation commands executed concurrently by the one of the services. This overwhelmed the system’s ability to process requests, leading to service disruption.

Mitigation

• We implemented rate limiting on module onboarding API calls (create, update, sync), reducing request frequency per pod, to ensure service availability.
• Increased CPU allocation for IaC server pods to mitigate auto-scaling cascades.
• Customers may notice that not all modules are created in a single run; reapplying the plan will complete the remaining modules.

Preventive and Long-Term Improvements

• We are evaluating a queued onboarding system to smooth out load spikes and avoid manual scaling coordination.
• Improvements in error handling and metadata synchronization logic are being addressed.