Overview
On November 7, 2023, a delay in the Autostopping feature of the Harness CCM platform was observed, affecting certain customers and causing Autostopping rules under fixed schedules not to start at the expected time. This issue was traced back to the system not correctly accounting for daylight savings time. The incident was resolved within 6 hours, with no actual downtime experienced by customers.
Timeline
Time
Event
November 7, 2023, 6:00 PM IST
Customer reported that Autostopping rules were not initiating as expected.
November 7, 2023, 6:27 PM IST
Incident response initiated.
November 7, 2023, 7:00 PM IST
Confirmed customers could manually start resources.
November 7, 2023, 7:00 PM IST
Issue identified - Fixed schedules were not accounting for daylight savings.
November 7, 2023, 8:00 PM IST
Ensured there were no infrastructure issues.
November 7, 2023, 8:30 PM IST
Generated new cron entries considering daylight savings.
November 7, 2023, 11:45 PM IST
Completed regenerating cron entries for all schedules.
Root Cause Analysis
The delay in Autostopping rules starting under fixed schedules was due to the system not accounting for daylight savings.
The issue was traced back through the following steps:
- Affected Autostopping rules were under fixed schedules with warm-up operations starting one hour before the expected time.
- The use of a time zone with daylight savings (America/New York) caused the schedule to start an hour earlier than expected.
- The initial generation of cron entries for schedules did not consider daylight savings.
- The system erroneously executed the idle time job because the schedule was triggered early, despite being under a fixed schedule.
Action Items
- The team is working on a long term fix for this so that daylight savings are automatically considered for fixed schedules
- Include the daylight computation savings in our tests plan