What was the issue?

Few customer pipelines with a high number of secrets were failing due to timeout errors.

Time (UTC) Event 11:21 AM UTC The issue started 11:21 AM UTC Alert for Name resolution delay error messages received. 11:35 AM UTC Team started investigating 11:41 AM UTC Incident was started 11:45 AM UTC Issue got resolved without any intervention at 11:41 AM UTC.

RCA:

Pipelines containing a large number of secrets, combined with system load, led to thread pool exhaustion during the decryption process. This caused timeouts for pipeline task execution. As the thread load decreased, the pipeline tasks recovered on their own.

Action Items:

• Thread Pool Optimization: Fine-tune the thread pool size to better match the current production load.
• Configurability: Make the thread pool size configurable to allow for gradual increases based on demand.