Prod2: Feature flag is currently degraded

Incident Overview

At 17:30 UTC, the FF streaming infrastructure experienced a 6x spike in simultaneous connection attempts across customers. This overwhelmed the feature flag service —which is responsible for maintaining long-lived streaming connections—and triggered repeated pod crashes due to memory exhaustion.

During the incident:

• Streaming was unavailable.
• SDK polling remained unaffected, allowing flag updates to still propagate.

Root Cause

The issue resulted from a convergence of two known risks:

1. Traffic Surge Loop: The volume of incoming /stream connection requests exceeded service capacity. Once pods restarted, SDKs immediately attempted to reconnect, triggering another wave of load, and creating a crash loop.
2. SDK Behavior (Node): A bug in Node SDK versions prior to 1.8.6 causes it to exponentially increase reconnect attempts when unable to reach the stream endpoint. This behavior amplified overall traffic, placing additional stress on the system.

Mitigation

• We immediately implemented rate limiting on the affected service to reduce saturation and break the crash-retry loop.
• Once stabilized, streaming service was restored within 30 minutes of mitigation.

Next Steps & Preventative Measures

• Rate Limiting Policy Enforcement: Permanent stream-level rate limiting is being added to protect the service from similar high-volume events in the future.
• Customer SDK Outreach: We are resuming targeted outreach to customers using legacy Node SDK versions with known reconnection issues to ensure timely upgrades.