Azure connectors failing for some customers

Incident Report for Harness

Postmortem

Issue Description: Customers experienced a disruption in the Azure copy of cloud cost data due to authentication failures caused by incorrect token formatting.

Resolution Time: 3 hours 48 minutes

Root Cause Analysis:

  • Token formatting had changed, leading to failures.
  • The token format changed because of a caching issue in the component responsible to populate the configmap which ended up in malformed configmap which is used in Azure data sync.
  • This was a partial outage impacting few customers.

Prevention Measures and Follow-Up Actions:

  1. Improve metric accuracy and alerting for datasync jobs.
  2. Enhance our automation suites.

Conclusion: The issue stemmed from token formatting inconsistencies, which have been addressed with updated tokens and preventive measures implemented to avoid future disruptions.

Posted Apr 19, 2024 - 03:07 PDT

Resolved

This incident has been resolved.
Posted Apr 08, 2024 - 13:00 PDT

Update

Cause of token failure found.
Posted Apr 08, 2024 - 12:08 PDT

Identified

New token was created. Rolled out in prod1 and 2.
Posted Apr 08, 2024 - 11:11 PDT

Update

FireHydrant incident created
Posted Apr 08, 2024 - 03:00 PDT

Investigating

Jira opened
Posted Apr 04, 2024 - 07:38 PDT
This incident affected: Prod 2 (Cloud Cost Management (CCM)).