Impact:

Some customers using the Harness CI cloud infrastructure pipelines failed with the following errors:

The delegate [dlite-dlite-free-xyz] disconnected while executing the task.

Who was impacted? Only Harness Cloud CI users whose pipelines were in a Running state during this incident.

‌

Root Cause:

The Harness Prod-2 service Dlite responsible for running CI tasks on Cloud VMs experienced a restart causing the running pipelines to fail.

‌

Detailed Timeline:

Time Event Aug 18, 2023 13:58 PDT Few Harness Cloud CI users whose pipelines were in a Running state started to fail Aug 18, 2023 14:20 PDT The issue was mitigated after restart of Dlite service which occurred due to the GCP automatic node upgrades. RCA process was initiated while monitoring continued Aug 18, 2023 17:20 PDT Root cause was identified Aug 18, 2023 18:00 PDT Incident was resolved

‌

Remediation Plan and Action Items:‌‌

• Why did the Dlite service on Prod-2 restart?

  • The restart of Dlite was triggered by the update of underlying nodes in the GKE cluster where it was deployed.

‌

• Why were the GKE nodes updated without notice?

  • The GKE nodes were updated simultaneously without any prior notice, as there was no maintenance window set. The simultaneous node update led to the "Dlite" service entering a degraded mode, causing failures in in-flight pipelines.

‌

• Why did all the nodes update simultaneously?

  • The Dlite service's deployment did not include a Pod Disruption Budget (PDB), which could have limited the impact of node updates by enforcing a rolling update strategy.

‌

• How can this be prevented in the future?

  • Set a maintenance window for Dlite and the rest of the CIE services similar to prod services
  • Set up Pod Disruption Budget (PDB) and Anti Affinity for all our services.
  • Tuning VM runner to be more resilient to restarts by de-centralizing the task assignments.