Custom Dashboards are not loading in Prod2

Incident Report for Harness

Postmortem

Summary 

On January 30th, TimescaleDB was running low on storage. To mitigate the issue, we began cleaning up unused indexes. However, one of the removed indexes was actively used by a custom dashboard, which subsequently led to degraded dashboard performance.

Resolution

In response, we promptly initiated an index rollback to restore dashboard performance and minimize customer impact.

RCA

One of the databases supporting custom dashboards was nearing its storage limit. To address this, we began reclaiming space through reindexing and applying data retention policies. During this cleanup process, a specific index - believed to be unused - was dropped to free up space. However, this index was actively used by the custom dashboard, leading to degraded performance.

Action Items

  • Implement automation for VACUUM and ANALYZE operations to ensure accurate index usage statistics, and establish a robust review process to validate dependencies before dropping any indexes. 
  • Plan and execute a database migration to a higher storage capacity.
Posted Apr 23, 2025 - 14:11 PDT

Resolved

This incident has been resolved.
Posted Jan 30, 2025 - 08:04 PST

Update

We are continuing to investigate this issue.
Posted Jan 30, 2025 - 07:15 PST

Investigating

Custom dashboards are failing to load in Prod2. We are currently looking into the issue.
Posted Jan 30, 2025 - 06:58 PST
This incident affected: Prod 2 (Custom Dashboards).