Summary

Due to a delayed background index build sync on the analytical node, a data replication lag was introduced. This lag prevented the latest pipeline executions from appearing in the Prod2 environment.

Resolution

Timeline (UTC) Event Oct 28, 9:09 AM The customer reported being unable to access the execution. Oct 28, 9:11 AM The team Engaged to troubleshoot the issue. Oct 28, 9:43 AM DBRE identified an issue with replication lag. Paused further index creation. Started bringing up new analytical node. Oct 28, 10:31 AM Index build sync completed and lag issue was resolved.

RCA

On Oct 28, 2024, between 9:09 AM UTC to 10:31 AM UTC, customers faced an issue where their recent pipeline executions were not visible in the prod2 environment. The issue stemmed from an index creation job executed as part of a recent release. This job caused replication lag in one of the read replicas of our MongoDB database, preventing up-to-date data from being available. The index creation job was halted, which restored normal replication and resolved the visibility issue for recent pipeline executions.

Action Items

Implement maxStalenessSeconds: - Application teams were advised to include the maxStalenessSeconds parameter in their connection configuration. This setting ensures read queries are directed to secondary nodes with replication lag below the specified maxStalenessSeconds threshold.