CCM Prod2 : Degraded Performance - Cluster Utilization and Cluster cost dashboards are not loading

Summary

Between Aug 5th at 10:40 AM PST and Aug 5th at 12:03 PM IST customers were unable to use the Cluster cost and utilization dashboards.

Root Cause

The team inadvertently introduced a bug during the implementation of a new Looker feature. This bug was caused by including a required_access_grant within a refinement.

Impact

Users encountered issues when attempting to access the Cluster cost and utilization dashboards.

Mitigation

• Deployed an updated build with fix.

Action Items

To prevent a recurrence of this incident, we will be implementing the following key actions:

• Further strengthen monitoring and alerting to detect early signs of similar issues.

• Implement preventative mechanism by enhancing the testing process. We are adding further code checks and increase coverage so that such issues do not occur.