Custom dashboards are not loading - Prod1,2,3,4 and Eu1 due to Looker managed service outage

Incident Report for Harness

Postmortem

Summary

Customers in all the production environments observed that custom dashboards were not loading correctly.

What was the issue?

Harness custom dashboards rely on Looker Studio, a managed service from Google. During that period, Looker experienced an outage, which directly affected custom dashboards.

Resolution

Once Google Looker was back to a stable state, custom dashboards started working correctly.

‌

Time (UTC)	Event
26 Mar 12:17 AM	Google Looker started experiencing an outage on login and dashboard functionality, impacting Harness custom dashboards.
26 Mar 1:29 AM	Google Looker managed service returned to a steady state.

‌

RCA

Custom dashboards lost availability due to an outage with Looker, a managed service from Google. No data loss was experienced.

Action Items

We are awaiting a follow-up RCA from the Google Looker team

Posted Mar 31, 2025 - 15:20 PDT

Resolved

The dashboards are now rendering correctly with the recovery of Looker Service.

Posted Mar 25, 2025 - 18:29 PDT

Monitoring

The dashboards are now rendering correctly with the recovery of Looker Service.We will continue to monitor the situation.

Posted Mar 25, 2025 - 18:23 PDT

Update

We are observing a gradual recovery of the Looker service. Some dashboards are now rendering correctly. However, a partial outage remains in effect. We will continue to monitor the situation and provide updates as they become available.

Posted Mar 25, 2025 - 17:56 PDT

Identified

Custom dashboards are not loading in Prod1, Prod2 and Prod3 because of our Managed service - Looker facing an outage currently. We will monitor this outage and update once we have a status update from Looker.

Posted Mar 25, 2025 - 17:17 PDT

This incident affected: Prod 4 (Custom Dashboards), Prod 3 (Custom Dashboards), Prod Eu1 (Custom Dashboards), Prod 1 (Custom Dashboards), and Prod 2 (Custom Dashboards).