Degraded performance in SEI insights

Summary

Few Efficiency and Productivity widgets were intermittently showing no data for few customers on Prod2 environment.

Root Cause :

SEI query engine service node had metadata sync issue with the rest of the cluster and was returning empty datasets. A defect in the service metadata sync process prevented the synchronization , resulting in query returning no data.
As queries are distributed across all nodes, the issue occurred intermittently—only when traffic was routed to the affected node. This made the problem appear random rather than systemic.

Impact :

For few SEI 2.0 customers, some Insights' widgets were impacted .

Productivity Insights : work completed, coding days per dev widgets were intermittently showing empty data.
Efficiency Insights : Lead time to change widget was intermittently showing empty data.

Duration:
February 23, 01:18 AM PST – February 23, 4:21 AM PST

What was not impacted?

The following systems and capabilities remained fully operational:

• Data ingestion and processing
• Integrations
• Org tree and other metadata operational flows
• All other efficiency, productivity, AI insights widgets
• Customers using SEI 1.0

No customer data was lost.

Remediation

• Immediate Mitigation: We identified a metadata synchronization issue as the root cause. We have implemented a fix that successfully stabilizes the system and prevents this bug from being triggered.

Action Items

To prevent such issues from happening again

• We permanently fixed the bug in downstream systems
• We are also enhancing and making our tests and alerting to make sure we detect this sooner.