Summary

Customers on Prod1, Prod2, and Prod3 clusters experienced intermittent widget load failures and increased load times when accessing AIDI 2.0 dashboards on July 22, 2026. Not all widgets were affected simultaneously the issue manifested as sporadic failures rather than a full outage.

No customer data was lost. SEI 1.0 customers were not impacted.

Root Cause

Over time, a routine database maintenance process failed to run on certain tables in our analytics database, causing those tables to accumulate a large volume of internal metadata used to track deleted records. When the database planned queries against these tables, it loaded all of this accumulated metadata into memory, causing memory usage on the affected nodes to spike repeatedly. These repeated spikes triggered an automatic safety mechanism that restarts a node when it detects excessive memory pressure, and the affected nodes began restarting in a loop as a result. This caused intermittent, degraded query performance on AIDI 2.0 dashboards for the duration of the incident.

Impact

Customers on Prod1, Prod2, and Prod3 clusters may have experienced intermittent widget load failures or increased load times on AIDI 2.0 dashboards.

Duration: July 22, 2026, 07:58 PDT – 16:16 PDT (~8 hours 18 minutes), with intermittent widget failures; system was restarted and under active monitoring from 08:25 PDT onward.

What was not impacted?

• Data ingestion and processing
• SEI 1.0 customers
• Integrations and metadata flows

No customer data was lost.

Remediation

Upon identifying the issue, the affected database nodes were restarted at 08:25 PDT, which restored initial stability. We continued to monitor the system closely, and when intermittent degradation was still observed afterward, we applied several additional fixes:

• Adjusted database configuration settings to limit the amount of memory used for processing accumulated metadata, and tuned query-planning settings to reduce memory pressure.
• Ran cleanup jobs to reduce the backlog of accumulated metadata on the affected tables.
• Increased capacity on the affected database nodes to provide additional headroom.

These changes progressively stabilized the system, and the incident was fully resolved at 16:16 PDT.

Action Items

To prevent recurrence, we are implementing the following:

1. We have upgraded the backend which includes underlying improvements that handle memory spikes caused by excessive delete files.
2. We have rolled out automated compaction jobs for newly introduced tables to prevent delete file accumulation going forward.