Impact:

• Custom dashboards in Prod 3 were not loading for accounts where the CI module is enabled.

‌

Root Cause:

• Why Custom dashboards were failing to load in Prod 3?
  Custom Dashboards are only available for Enterprise plans on Harness. The custom dashboard micro service migrated to leverage module licenses instead of module feature flags, CI team deprecated the module-enabled FF. The code changes to handle feature flag deprecation in the dashboard service were available in versions v1.53.18 and higher, due to version mismatches there was a misalignment in contracts.

‌

Timeline:

Time Event Jul 14, 12:32 PDT Internal incident call triggered and investigation started. Jul 14, 12:51 PDT Identified the root cause, triggered a deployment to bump the version of custom dashboards micro service. Jul 14, 13:13 PDT Custom Dashboards functionality issues were addressed and normal operations resumed

‌

Remediation Plan and Action Items:

• Make custom dashboards micro service agnostic to the versions of other Harness microservices, all the changes have already been made on versions v1.53.18 and higher.
• Setting up Pingdom alerts for custom dashboards endpoints on the portal to proactively capture and get alerted on issues.
• Leverage Prometheus Metrics to expose custom metrics in order to proactively capture and resolve such issues going forward.