On November 13, 2023, a failure in CCM Cloud Connectors impacted customers with currency preferences enabled. The external API responsible for fetching currency rates failed, resulting in data ingestion failure. This incident solely affected data ingestion, and all other features of CCM remained unaffected. The incident was successfully resolved within ~2 hours and 15 minutes, with no reported downtime.
| Time | Event |
|---|---|
| 2023-11-13, 02:15 PM UTC | Issue first reported on slack channel for a customer account |
| 2023-11-13, 02:30 PM UTC | Incident acknowledged, and internal investigation initiated |
| 2023-11-13, 03:17 PM UTC | Root cause identified |
| 2023-11-13, 03:29 PM UTC | Temporary fix raised, followed by deployment of latest code |
| 2023-11-13, 06:55 PM UTC | Data replayed for all affected customers |
The incident originated from the failure of an external API fetching currency rates, impacting data ingestion for CCM Cloud Connectors with currency preferences. The decision to use an external API for currency rates was necessitated by the dynamic nature of currency conversion rates. The incident was further exacerbated by the failure of the fallback mechanism, backup currency rates were not populated for the current month.