CI/CD Pipeline failure on Prod4

Summary:

A Deployment pipeline execution on Prod4 resulted in removal of few workload identities from DR cluster which are shared across Primary and DR clusters, this caused the pods in both the primary and DR clusters that depend on workload identity to fail, affecting service availability.

What was the issue?

Customers faced issue with their CI/CD Pipeline where the pipeline started failing on Prod4environment

Timeline:

Resolution:

Re-deployment resolved the issue by ensuring that the Terraform state was aligned with the intended configuration, which restored the missing workload identity bindings in the DR cluster.

RCA

During a recent Disaster Recovery (DR) exercise, manual changes were made to the DR cluster but were not properly captured in the Terraform configuration. When the deployment pipeline executed, Terraform applied its last known state, inadvertently removing the workload identity bindings in the DR cluster. This led to pod failures in both the Primary and DR clusters, causing the CI/CD pipeline to fail in Prod4.

Action Item

1. Ensure no manual changes are made to the system. In case of unforeseen manual changes, document them and incorporate them into Terraform.
2. Automate Drift Detection: Implement automated drift detection to identify discrepancies between the live infrastructure and Terraform state.
3. Pre-Deployment Validations: Introduce additional pre-deployment checks to verify workload identity bindings before applying changes.