Overview

On November 7, 2023, a delay in the Autostopping feature of the Harness CCM platform was observed, affecting certain customers and causing Autostopping rules under fixed schedules not to start at the expected time. This issue was traced back to the system not correctly accounting for daylight savings time. The incident was resolved within 6 hours, with no actual downtime experienced by customers.

Timeline

‌

Root Cause Analysis

The delay in Autostopping rules starting under fixed schedules was due to the system not accounting for daylight savings.

The issue was traced back through the following steps:

1. Affected Autostopping rules were under fixed schedules with warm-up operations starting one hour before the expected time.
2. The use of a time zone with daylight savings (America/New York) caused the schedule to start an hour earlier than expected.
3. The initial generation of cron entries for schedules did not consider daylight savings.
4. The system erroneously executed the idle time job because the schedule was triggered early, despite being under a fixed schedule.

Action Items

1. The team is working on a long term fix for this so that daylight savings are automatically considered for fixed schedules
2. Include the daylight computation savings in our tests plan