We want to share the details about the slowness observed in our Prod2 cluster that impacted our customers on 12/22 between 3:41 AM — 7:14 AM PST.

‌

Impact:

Harness manager UI was slow intermittently during the interval of 3:41 AM and 5:55 AM PST when the major GC was running on the pods.

‌

Root Cause:

The issue was happening due to a script execution which was listing pipelines across all projects by an internal user, which led to a large number of account objects in a heap. This script caused the memory load on the system, and can only happen if a user is an internal Harness user due to available access to many accounts.

‌

Incident timeline:

3:41 AM - Pod restart alerts for CG Manager in internal slack alerts channel

3:46 AM - Internal zoom call started.

3:54 AM - Another set of alerts for pod restarts and ART alerts came. The incident action team started FireHydrant, and the status page was updated.

4:04 AM - Increased the pod memory to 40 GB from 24 GB. (It was reduced with recent deployment, so we reverted that change to rule out that possibility), with this ART settled down.

4:13 AM - Heap utilization alert came. And we noticed Major GCs in the AppD dashboard. The team was already looking at Heap Dump. Only one of many pods had a Heap dump available, investigation started on why all pod heap dumps did not get captured. CPU utilization was going up as well for CG Manager.

4:24 AM - Deployment rollback was initiated to rule out application issues due to today’s deployment.

4:53 AM - Issue was marked mitigated. Post-prod sanity test was done.

5:21 AM - Alerts happened again.

5:45 AM - Reduced Manager Iterator to 1 instance from 2 to rule out contention issues, and alerts stopped again.

5:55 AM - The last alert came.

6:10 AM - Heap histogram analysis indicated a large number of account objects in a heap, and the corresponding thread dump indicated it was due to the following GET /api/ng/user/internal_account_id call. This happens in validate token call, we reached out to the script owner for this internal account.

6:40 AM - Status page was updated to Monitoring.

6:45 AM - With further analysis, it was established that this was happening due to running a script to list pipelines across all his projects which was the prime suspect at this point.

6:54 AM - Team root caused the script that caused the memory load on the system, and we closed the internal incident as mitigated. This can only happen if a user is an internal Harness user since we have access to a lot of accounts.

7:14 AM - Status page was marked as Resolved

‌

Remediation:

• As an immediate remedy, as the scope was narrowed down to internal user making API calls through the script, we have refrained from executing the script for now.

‌

Action items:

• We are blocking all internal users from pulling any data across accounts via API so that we won't see this issue again
• Hotfix Manager/NG Manager to not load the support accounts in API-key validation.