Impact:
The impact was only in Prod 2 cluster and due to this below issues were occurring
- Inability to start or stop resources based on idleness detection.
- Disruption of scheduled tasks and routine health check jobs.
- Impaired functionality for executing schedules and related tasks.
Root Cause:
Due to memory allocation issues, the background job processing system stalled, resulting in auto-stopping services facing partial outage as the auto-stopping jobs got lined up in the queue and could not be processed.
Detailed Timeline:
Time
Event
Jun 28th 12:10 PM PDT
Initial report of some of the AutoStopping background operations such as Load Balancer update showing signs of slow response
June 28th 5:00 PM PDT
The background job system restarted
June 28th 5:05 PM PDT
The background job system resumed processing and the backlog started clearing up
June 28th 7:30 PM PDT
Resource usage started increasing and more restarts of the background job system observed
June 28th 8:00 PM PDT
Started investigating why the background operations are not working
June 28th 8:45 PM PDT
Identified that Background job system were unstable and workers were not able to get jobs from it
June 28th 9:00 PM PDT
Re-deployed Background job system after correcting memory requirements
Remediation Plan and Action Items:
- Monitoring mechanisms are set to observe the resource consumption of the background job processing system.
- Alerting mechanism setup to notify the Harness team when the jobs from the queue are getting processed slowly.