Summary
On February 11th, 2026, Harness Dashboards experienced an issue with data staleness within the prod3 environment. During this period, newly added data was not reflected in the dashboards.
Root Cause
We had some unexpected spike in load which caused resource exhaustion. This interruption prevented new data changes from being reflected in the replicas that power the dashboards.
Impact
Customers were viewing stale data, as no new changes were visible in the dashboards and approximate duration of impact was from 12:30 PM PST to 05:30 AM PST. The issue was limited exclusively to the dashboards and all other functionality was working as expected. No customer data was lost or damaged. The issue only affected data visibility on the dashboards during the specified timeframe.
Mitigation and Permanent Fix
To mitigate, we increased capacity to effectively manage and to handle the spike in workload.
Action Items
To prevent such issues from happening further we are enhancing Monitoring & Early Detection by Adding replication lag alerts with severity thresholds. We will also proactively make the system resilient and robust by running performance tests and fine tune the system.