Impact:

On Friday, September 15th, for a short period of time between 11:12 PDT and 11:25 PDT, customers started receiving 503 error codes in response to some SDK authentication and evaluation requests. The users were not able to toggle flags via the admin user interface during this time.

‌

Root Cause:

The incident occurred following an upgrade to Redis v6 from Redis v5.

The Redis instance CPU spiked at 100%, this was due to one command that execute in O(N) time complexity. As the number of keys in the database grew during the day, the commands took longer to execute. As Redis v6 was able to execute more commands in parallel the CPU became maxed out resulting in 100% usage across all cores.

The impact was that queries took so long, that feature flag requests started to time out.

‌

Detailed Timeline:

Time Event Sept 15th, 2023 11:12 PDT First alert came through the ff-alerts channel, with ab Uptime check failed ff-admin-server Pods started to restart in Prod 2 ff-client-server Pods in a not ready state SRE jumped on to start debugging the issue Quickly identified Redis CPU being at 100% for a period Looking at the logs of the pods that were restarting, error message indicating the ff-admin-server was unable to connect to the Redis instance Scaled down the Pods for the ff-admin-server and ff-client-server deployment to 2 to reduce load on Redis Once the Redis CPU dropped from 100%, pods brought back up to previous levels Sept 15th, 2023 11:25 PDT Last Failed health check reported Sept 15th, 2023 11:27 PDT Incident raised as a FH Sept 15th, 2023 11:27 PDT Manual testing on Prod2 demo app, works as expected Sept 15th, 2023 12:00 PDT Decision to go back to the older version of Redis, until further investigation can be performed. Old Redis instance was still kept around after the upgrade, so only needed to do a config change, and redeploy the affected services

‌

Remediation Plan and Action Items:‌‌

The RCA below has been split into two.

1. Why is the feature flag admin service restart preventing users from toggling flags?
2. Why SDK requests failed with 503.

‌

Admin Server Restarts

Why 1 - the ff admin service started to restart

The admin services liveness probe started failing. As a result, Kubernetes started to restart the service.

Why 2 - the liveness probe started to fail

The probe uses the admin services health-check endpoint. This checks that all downstream services are available and responding. Requests to the Redis service had started to timeout.

Why 3 - the Redis health check started to fail

The Redis service was reporting 100% CPU usage. While it was processing commands there was increased latency which resulted in many calls timing out.

Why 4 - did Redis CPU spike at 100%

We recently upgraded from Redis v5 to Redis v6. Redis v6 is multi-threaded and can perform operations concurrently.

Why 5 - did running more commands concurrently increase CPU usage to 100%

One of the Redis calls used by Feature Flags CDN solution is called ‘keys’. This returns keys based on a pattern match. Although it uses a pattern match in the command, the execution time for the command increases linearly as the total number of keys in the cache increases. During the peak part of the day, as the number of keys in the cache grew, it reached the point where the command took longer than the configured timeout to return.

‌

Evaluation Service Failures

Why 1 - did the evaluation service start returning 503 errors

The logs indicated requests were starting to timeout. As a result, a 503 was returned.

Why 2- did requests timeout

During the authentication and evaluation flows Feature Flags makes use of Redis to cache data. This allows us to scale the service. Some of the Redis queries started to timeout.

Why 3 - did Redis call time out

The Redis service CPU had spiked at 100%. Normally if we cannot find an item in the cache we fall through to the underlying database, but in this case because the request timed out we failed to hit the underlying database to load the data.

Why 4 - did Redis CPU spike at 100%

(see why 4 from the admin service failures)

Why 5 - did running more commands concurrently increase CPU usage to 100%

(see why 5 from the admin service failures)

‌

Why was this not identified during the testing

There are two use-cases load tested prior to release

• Serving requests from CDN

The part of the solution that used the expensive Redis calls occurs during requests that are served out of the CDN to guarantee the requested ETag is still up to date. This typically operates over a small set of keys, so when load testing only a small set of a few thousand keys are used.

‌

• Performing live evaluations

When testing evaluation calls, this normally has a cache with a few hundred thousand keys. But this flow does not use expensive Redis calls.

These flows are tested in isolation in our testing environment, but the issue only manifests when combining the two scenarios.

Follow up actions

• Update the admin server k8 probe. It should not trigger restarts as a result of a downstream service.
• Replace the Redis ‘keys’ command with a call that completes in O(1) time. This ensures consistent response times regardless of how big the cache grows
• Review the architecture. If the Redis call times out the overall requests fail, we should still be able to fall back to the underlying database, although this will increase the response time it will allow the solution to continue to serve traffic even while some services are suffering a complete outage.