Impact

The Feature Flags UI stopped loading for 1 hour and 10 minutes, and customers were unable to manage their Feature Flags via the UI. Neither flag evaluations or SDK communication was impacted.

Root Cause

A health check that runs against our database was unable to resolve due to a query logjam caused by Git-sync related connections not exiting properly.

This in turn caused pods in our Feature Flag admin service to report as unhealthy and stop resolving.

Timeline

Remediation

For immediate resolution, we scaled up the Feature flag admin service, which immediately created healthy new pods and restored functionality.

Action items

1. Address the code that can result in queries/transactions not closing during an error state. The code should be wrapped in a defer block.
2. Update the health check probe. Currently it checks if a single API returns OK, but we can include stats to see if our DB response times are > than x instead. Those failures can trigger a restart which would remediate the initial issue.
3. More tightly coupling alerts for this service with our incident management process so that going forward incidence instigation will be much prompter.