PROD2: Delegates got disconnected from Harness

Summary

A subset of Delegates in prod2 cluster got disconnected, causing pipeline failures for customers. It was due to an increased load on the backed database due to an ad-hoc read query.

What was the issue?

Customer delegates were disconnected and pipelines were failing.

Resolution

We cancelled the runaway query, and upscaled the database. Overall recovery took ~17 minutes, and the majority of Kubernetes delegates reconnected automatically. A few of the customers had to restart the non-kubernetes delegates.

RCA

As part of a regular operational work, we ran a read query in the database which spiked the CPU usage on the database. Unfortunately, this query was run against the primary replica, which increased query latency, resulting in some delegates getting marked disconnected.

Action Items

1. Enhance access control:  We have a Just-In-Time read access to our database for operational tasks. We are enhancing our system to only provide access to non-primary replicas for such operations.
2. Enhanced resiliency: We are planning to run chaos experiments simulating db latency to improve resiliency in our delegate management sub-system against such faults.