On 08 May 2025 at 4:05 AM UTC, a system release introduced a change that led to increased data ingestion activity. As a result, customers experienced latency in the Unified Pipeline dashboard causing temporary data staleness within the Prod3 environment.
There was no downtime or data loss, but dashboard visibility was temporarily impacted. The issue has since been mitigated, and preventive measures are being implemented to avoid recurrence.
A system release triggered high-frequency data ingestion, which caused processing delays and led to temporary latency in the Unified Pipeline dashboard within the Prod3 environment.
| Timeline | Activity |
|---|---|
| 08 May 2025 at 4:05 AM UTC | Prod3 system release completed |
| 08 May 2025 at 12:17 PM UTC | Issue was identified |
| 08 May 2025 at 01:48 PM UTC | A fix to increase system resources of our databases was validated and implemented |
| 08 May 2025 at 3:20 PM UTC | Data was caught up and issue resolved |
To mitigate the processing delays, the database resources were scaled up. This scaling operation helped restore normal processing throughput. Once completed, the latency issue in the Unified Pipeline dashboard was resolved, and full visibility was restored in the Prod3 environment.
Move major data migrations under a feature flag and schedule them over weekends to ensure better processing performance and minimize impact on live systems.