Overview

TI service received burst cleanup calls at two different times and the cleanup procedure created a huge load on timescale DB used by TI service. The CPU of timescale was running at 100% due to which ping to timescale was failing (TI service readiness probe). Since TI service readiness probe was failing, Kubernetes marked the service pods unavailable.

Impact

CI pipelines executions which upload test reports and were started during the below time window may have been impacted.
Times: (2:15 AM - 2:50 AM and 4:45 AM to 5:15 AM) PT

Resolution

Cleanup requested that were queued got completed and system returned to operational

Timeline

Time Event Notes 10/16/2023 02:16 AM TI service received burst cleanup requests 4:44 AM TI service received burst cleanup requests 6:28 AM TI service 503 error reported 7:45 AM Discovered that timescale was running at 100% 8:00 AM Suspected that cleanup jobs were hogging the timescale CPU Platform team confirmed that data deletion job was run due to a bug Still going through TI service logs to find any other suspects 8:30 AM Correlated and confirmed that the cleanup led to high timescale CPU Both the readiness probe failures times coincided with the cleanup job requests 9:00 AM Incident was resolved Action items were discussed for Engg

‌

RCA

Timescale was not responding to pings which is the readiness probe for TI service. Since TI service readiness probe was unresponsive, Kubernetes marked the service pods unavailable. Timescale DB was running at 100% CPU due to high number of procedure calls made from TI service for periodic cleanup. The DB was already under high utilization due to production workload since all the test reports are stored in this DB. The CG Manager code which runs multiple threads in parallel for periodic cleanup ran on a weekday (supposed to run on weekends) due to a bug which sends cleanup events to all services. CI Manager picked up these events and sent burst cleanup API calls to TI service.