Incident Summary:
Due to an increase in traffic, there was a period of high latency experienced on the Feature Flag metrics service
This was caused by the service not able to scale up quickly enough to handle the additional load automatically, and the service to become slow, and returning errors.
Once identified by the team, the cloud engineer was able to manually scale up the service, and the service was restored
Timeline
| Time (UTC) | Event |
|---|---|
| 18:11 PM | Large number of requests seen coming through the network |
| 18:14 PM | Service gets into a degraded state, returning an increase in errors, and latency |
| 18:14 PM | On call engineer is alerted and begins investigation |
| 18:24 PM | Service is manually scaled up to handle the load |
| 18:24 PM | Development team begin RCA |
| 18:41 PM | All requests return to normal operational behaviour |
| 18:41 PM | Incident resolved |
Root Cause Analysis:
The incident originated from an increased rate of requests on the Prod 1 environment, causing the Feature Flag metrics service to get into a degraded state.
While the service has auto-scaling capabilities in place, the sudden increase, and size of the increase resulted in the automated scaling to be inefficient, and manual intervention was required
Immediate Resolution:
To address the incident promptly, the team increased the resource capacity of the affected service, until the service was able to resume normal operations.
Preventive Measures:
To prevent similar incidents in the future while the team are addressing working on improvements, resources have been adjusted in the affected cluster to better handle sudden traffic spikes
Action Items:
We have identified a number of bottlenecks that resulted in the incident, and the development team are actively working on improvements