Software engineering insights dashboards are unavailable

Incident Report for Harness

Postmortem

Summary

On February 4, 2026, SEI experienced a temporary outage that affected SEI Insights. SEI Customers (US production clusters) were unable to see Insights data during this period. Customer data remained safe and was not impacted.

Root Cause

An automated system update caused our main database service to stop working correctly. When it restarted, it did not recover as expected, which led to temporary service errors.

Impact

  • Customer impact: SEI Insights did not display data
  • Duration: Approximately 9:33 AM to 2:37 PM PST
  • Scope: SEI Insights Only
  • Data safety: No customer data was lost or damaged

Mitigation and Permanent Fix

To restore service, we created and switched to a new, more resilient database setup. This allowed SEI Insights to return to normal operation.

Next Steps

  • Keeping a backup system ready to reduce downtime if a similar issue occurs
  • Closely monitoring the new setup to quickly detect and address issues
Posted Feb 10, 2026 - 08:32 PST

Resolved

This incident has been resolved.
Posted Feb 04, 2026 - 15:21 PST

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Feb 04, 2026 - 14:34 PST

Update

We are deploying the fix to all clusters now. Starting with production -1 environment
Posted Feb 04, 2026 - 13:41 PST

Update

We are working towards the fix and are verifying this in a lower environment.
Posted Feb 04, 2026 - 11:34 PST

Update

We have identified the database component that caused the issue and are working to restore it at the moment.
Posted Feb 04, 2026 - 10:46 PST

Identified

The issue has been identified and team is working on a fix. The issue started at 8:15 PDT
Posted Feb 04, 2026 - 09:41 PST

Investigating

We are currently investigating this issue.
Posted Feb 04, 2026 - 09:38 PST
This incident affected: Prod Eu1 (Software Engineering Insights (SEI)), Prod 1 (Software Engineering Insights (SEI)), Prod 2 (Software Engineering Insights (SEI)), and Prod 3 (Software Engineering Insights (SEI)).