Data delay in Custom Dashboards

Summary

‌

On August 28 customers using deployment dashboards in Production encountered “data staleness”. No data loss occurred. All other product areas remained fully functional.

Root Cause

The ETL service that powers these dashboards had a memory spike which paused further processing of raw pipeline data.

Impact

‌

Deployment dashboards served stale data via custom dashboards. 

Remediation

• ETL service was restarted with bigger memory allocation to avoid future out of memory errors.

Action Items

Enhance monitoring & alerting – Add monitoring on ETL service to alert when a certain threshold of memory is breached and update capacity if needed.