PROD1: Unified Custom Dashboards are not loading properly

Incident Report for Harness

Postmortem

Summary

On April 8, 2025, for 25 minutes, customers in the prod-1 production environment observed that the following custom dashboards were not loading properly: pipeline, stage, and step executions. We discovered that necessary model changes were missed during the version upgrade of our ETL process.

Resolution

Upgrading the ETL process to a newer version addressed this issue.

RCA

Pipeline, stage, and step execution custom dashboards were not loading correctly due to an incorrect upgrade of the ETL process. The incorrect upgrade resulted in our views not having the necessary data to render the dashboards. While no data loss was experienced, dashboards were not rendering correctly for a brief period.

Action Items

Improve Pre-Deployment Checks for ETL service upgrade: Enhance pre-deployment checks to validate critical model updates are part of the upgrade process.

Posted Apr 23, 2025 - 13:12 PDT

Resolved

We have resolved the issue. Dashboards are up and running.

Posted Apr 08, 2025 - 12:38 PDT

Identified

We have identified a potential cause of the service issues and are working hard to address it. Please continue to monitor this page for updates.

Posted Apr 08, 2025 - 12:14 PDT

Investigating

We are currently investigating this issue. This is impacting steps, stages and pipeline execution dashboards.

Posted Apr 08, 2025 - 11:37 PDT

This incident affected: Prod 1 (Custom Dashboards).