Summary:

Customers in Prod3 were unable to run Hosted CI builds.

What was the issue:

The delegate lite microservice that is responsible for hosted CI builds got scaled down due to a misconfiguration so the customers were unable to run CI builds.

Resolution

Time Event Dec 10 at 5:34 AM UTC Customer reported an issue with running CI pipeline. Dec 10 at 5:52 AM UTC Redeployment of Delegate lite microservice was done in Prod3 and the issue resolved.

RCA

The delegate lite deployment follows a blue green deployment model. However, due to a misconfiguration, both the old and the new deployment were stopped after the maintenance window, which was intended to shut down only the old deployment. The issue did not get detected internally due to a misconfigured alert.

Action Items

• Fix and test the alerts for Hosted CI and related services to ensure they trigger as desired.
• Fix the delegate lite deployment to scale down only if the required number of containers are active.