Summary
On June 4, 2026, customers using Harness FME SDKs experienced degraded streaming performance. SDK connections repeatedly disconnected and fell back to polling mode, resulting in delayed flag propagation and elevated log noise. Flag evaluation itself was unaffected — all SDKs are designed to poll against the CDN during streaming disconnects.
Impact
- Region affected: us-east-1 (other regions — EU, APAC — remained healthy)
- Customer impact: SDK streaming connections intermittently dropped, causing SDKs to fall back to polling. Flag evaluation continued uninterrupted; customers experienced delayed flag propagation
Root Cause
The underlying cause was memory exhaustion in the backend messaging cluster in the us-east-1 region, triggered by a combination of an ongoing service rollout and a traffic spike during the same . Other regions remained fully healthy throughout.
Remediation Actions Taken
- Service was rolled back to the previously stable version.
- Increased capacity to handle the surge of connections.
Preventive Measures / Next Steps
To prevent such issues happening again, Harness is working on several optimizations:
- Enhance deployments to do validation before broader rollout to catch capacity pressure early.
- Improve connection handling during deployments or more conservative deployment pacing during high-load periods.
- Improved SDK behavior on reconnect: Connection recycles inherently generate a resync storm; further investigation ongoing into reducing the cascading request impact from mass reconnects