Prod3 experiences slowness in pipelines

Summary

A rollout involving OpenTelemetry instrumentation changes introduced a memory leak in the OTEL eBPF collector running in production clusters. Under sustained production traffic, the leak caused increasing JVM heap utilization, elevated garbage collection pressure, and eventual out-of-memory (OOM) conditions across several core platform services.

Impact

• Elevated latency and intermittent instability in Prod1, Prod2, and Prod3
• Some customers experienced slow pipeline execution and degraded responsiveness

No customer data loss occurred.

Root Cause

The root cause was an upstream defect in the OpenTelemetry eBPF instrumentation library that introduced a memory leak under production-scale workloads. The leak continuously increased telemetry-related memory consumption, leading to sustained JVM garbage collection pressure and eventual heap exhaustion.

Mitigation and Recovery

Immediate Actions

• scaled up clusters to stabilize impacted clusters
• Disabled OTEL instrumentation components and restarted affected services

Next Steps

To prevent such issues from happening again, we are:

• Enhance our load testing process to test in higher workloads to identify such issues prior to going production.
• Add additional granular instrumentation to catch such issues sooner.