Chaos Engineering users in Prod2 cluster may be experiencing issues

Incident Summary:

The Chaos dashboard on the Prod2 cluster was inaccessible to users, returning a 404 error code when attempting to access it.

Timeline

Root Cause Analysis:

During the Helm switchover, all module ingresses were duplicated under a new Ingress Class. Automation was created to ensure that each service's ingress rule was duplicated. However, the Chaos ingress rules followed an older method of implementing the Ingress Class which uses annotations, which was missed by the automation.

Immediate Resolution:

The Chaos ingress rules were recreated with valid Ingress Class.

Action Items:

• Update the Chaos ingress rules to use the new method of associating the Ingress Class using Key Value Pair.
• Enhance the post-migration validation framework to include automated checks for data accuracy and integrity.