Customers experienced pipeline failures due to intermittent errors when submitting delegate tasks. The issue was identified by the error message:
UNAVAILABLE: Connection closed after GOAWAY. HTTP/2 error code: NO_ERROR, debug data: max_age
Timeline
| Time (UTC) | Event |
|---|---|
| 06:00 AM | First occurrence of the issue. |
| 06:06 AM | Alert from our monitoring system received; team started investigating. |
| 06:42 AM | Service instances scaled up to restore service. |
| 07:00 AM | Functionality enabled (which was already being rolled out behind a Feature Flag) to prevent the reoccurrence of the issue. |
RCA
Pipeline execution functionality was degraded due to exhaustion of thread pool resources (responsible for secret resolution from custom secret manager). Trigger was pipeline run with a large number of secrets, which overwhelmed the thread pool responsible for resolving secrets. This reduced the capacity of the system, resulting in a build-up of delegate tasks awaiting submission. Eventually, those requests timed out, leading to pipeline failures.
Once issue got identified, we immediately scaled up our service infrastructure to handle the increased load. Subsequently, a feature flag to optimize secrets resolution flow was enabled. (This feature flag was in process to be enabled across all Harness environments in the next few days).
Actions Items