Impact:

• The Pipelines which were using the templates failed to load and execute
• Templates were not loading for any customer
• Other pipelines (not using templates) were working fine

‌

Root Cause:

• Why Pipelines(using templates) and templates were failing to load in Prod 1?
  It was due to template service which was unstable and taking too long to respond.
• Why was the Templates service unstable and responding slowly?
  Template service was getting restarted due to OOMs (since pipeline services depend on template service to get resolved template service, it was also not usable)
• Why there were OOMs in the Template service?
  A certain case surfaced the circular dependency between template service and ng-manager service. This led to a recursive loop which drastically increased the number of calls resulting in an increase in memory and resulting into OOMs

‌

Timeline:

Time Event June 7th 8:14 PM IST Internal incident call triggered and investigation started. June 7th 8:27 PM IST Template service started crashing with OOM June 7th 8:37 PM IST Identified that one customer is using custom secret manager whose template is stored in remote. This is causing recursive calls leading to service crash. Increased memory and heap size of template service also bounced template service pods which mitigated issue temporarily and investigation continued. June 7th 8:50 PM IST We again started seeing CPU spikes and pods health checks started failing. Reached out to affecting customer. June 7th 9:00 PM IST Blocked traffic to the specific project via Cloud Armor rules June 7th 9:15 PM IST Bounced template service pods which mitigated issue temporarily and investigation continued. June 7th 9:25 PM IST We again started seeing CPU spikes and pods health checks started failing. June 7th 9:45 PM IST Blocked the pipeline via GCP Cloud Armour for the customer and also disabled related triggers via DB updates. June 7th 10:05 PM IST Bounced template service pods which mitigated issue temporarily and investigation continued. June 7th 10:12 PM IST We again started seeing CPU spikes and pods health checks started failing June 7th 10:15 PM IST Template service replica was set to zero June 7th 10:22 PM IST Template service brought back to original replica. June 7th 10:25 PM IST No more issue spike observed. Teams started sanity. June 7th 10:38 PM IST Notified services were back to normal and incident was resolved.

‌

Remediation Plan and Action Items:

• Short term:

  • We blocked the impacting customer's pipeline executions at GCP Cloud Armour
  • Removed the option from UI to move the related secret manager template to remote.

• Medium-term:

  • Verify for each type of template if the remote option needs to be supported.
  • Remove the remote template option for Custom Secret Manager & Artifact Source type of templates

• Long term:

  • We need to restrict the impact of such circular dependency issues only to one account and not to the whole cluster.
  • Check for any other place this circular dependency can happen and have architectural guidelines so such scenarios never happen again.