Impact:

Users in Harness Next Generation platform in the Prod 1 cluster noticed slower than normal service:

1. The UI calls were slow/failing making UI unusable
2. The role bindings created after incident did not get processed and hence would not take affect
3. We checked various customer’s pipelines during this time and we can confirm that they were running smoothly and there was no impact on executing pipelines

Root Cause:

Access control service was doing major GC, because of which the health checks were failing and service was restarting. It was loading up millions of ACLs in memory for processing the role binding created by a specific unusual request. The logic currently creates all the ACLs in memory before pushing them in db in batches.

Timeline:

May 2nd, IST

Time Event 3:30 PM A customer created a complex role binding 4:15 PM First devops alert came from appdynamics on ART being high 4:30 PM Internal call started and we determined that the access control service is doing full GCs and we determined to increase memory 5:15 PM access control with 6gb heap and 8 gb pod size was rolled out, that did not help the situation 5:35 PM access control with 12gb heap and 16 gb pod size was rolled out, that stabilized the access control service

Remediation:

Increasing pods resources to prevent OOM and pod restart.

Action Items:

• Hotfix to save ACLs in batches instead of loading in memory.
• QA setup to add custom role binding scenarios