CCM Perspectives and Dashboards are not working

Incident Report for Harness

Postmortem

Overview

All customers experienced disruptions with Dashboards and Perspectives. However, users could still access CCM and utilize other sections of the system.

Timeline (PST)

Time	Event
10:17 AM Sept 28 2023	Users reporting slowness in Dashboards and perspectives. Engineering team started the investigation.
11:46 AM Sept 28 2023	Finds out GCP Bigquery is having autoscaling issues for muti-region US.
12:52 PM Sept 28 2023	The usage of BigQuery has been redirected to a different service account from another GCP project that employs on-demand pricing.
~2:30 PM Sept 28 2023	Perspectives have been restored and are now operational. Dashboards are functioning to some extent, though a few customers are still experiencing issues.
5:12 PM Sept 28 2023	Google reports BigQuery issue resolved
5:12 PM Sept 28 2023	Dashboards have been fully restored and are now operational for all customers.

Resolution

Redirecting BigQuery's service account to a different GCP project with on-demand billing resolved the problem.

Affected Users

Users in Prod1 and Prod2 were impacted. Only the Perspectives and Dashboards features were affected, while the rest of CCM operated without issues.

RCA

Google's BigQuery employs slot AutoScaling to enhance slot availability for better performance. An incident with BigQuery hindered the slot AutoScaling functionality. Since CCM's perspectives and dashboards rely on BigQuery, the incident impacted their query response times.

Posted Oct 24, 2023 - 13:02 PDT

Resolved

This incident has been resolved.

Posted Sep 28, 2023 - 17:36 PDT

Update

We are continuing to monitor the workaround in place for any issues. Additionally, we are seeing responses come back from our primary upstream provider but are awaiting additional status update/confirmation from the provider(GCP) before affirmation.

Posted Sep 28, 2023 - 16:33 PDT

Monitoring

Harness has recently implemented a workaround for the majority of customers which should restore CCM Perspectives as well as Custom Dashboards. Our primary upstream provider(GCP) however, while showing some signs of recovery is still encountering issues. Harness will continue to provide status updates as we closely monitor the workaround we have put in place to mitigate issues related to CCM Perspectives and Dashboards until the incident is fully resolved.

(Note: Existing GCP/CCM connector validation/test functionality will error out due to the workaround in place, but does not affect product functionality)

Posted Sep 28, 2023 - 14:51 PDT

Update

Harness has recently implemented a workaround for the majority of customers which should restore CCM Perspectives as well as Dashboards. Our primary upstream provider is still experiencing issues and is showing signs of recovery, and we expect an update within the next 30 minutes to the upstream status.

Note: Existing GCP/CCM connector validation/tests will error out due to the workaround in place.

Posted Sep 28, 2023 - 14:39 PDT

Identified

We identify that the issue we are experiencing is related to the Google Big Query incident:

https://status.cloud.google.com/incidents/V8br4RDzg1RsCw6zWQEv

Posted Sep 28, 2023 - 13:53 PDT

Investigating

CCM Perspectives and Dashboards are down because of a GCP Bigquery incident. We are currently investigating this issue.

Posted Sep 28, 2023 - 13:23 PDT

This incident affected: Prod 1 (Cloud Cost Management (CCM), Custom Dashboards), Prod 2 (Cloud Cost Management (CCM), Custom Dashboards), and Prod 3 (Cloud Cost Management (CCM), Custom Dashboards).