Summary
Between 21:31 UTC and 21:56 UTC on October 31, 2025, customers using Prod1 experienced failures when submitting support tickets through the Zendesk integration and encountered a generic “Something went wrong” message. Prod2 and Prod3 remained fully operational. The issue was root caused to a configuration issue.
Root Cause
A recent migration caused Prod1 to miss the required Zendesk configuration. This missing configuration prevented authentication with Zendesk, blocking ticket creation for users in that environment.
Impact
- Customer Impact: Customers were unable to create support tickets in Prod1.
- Other Environments: Prod2 and Prod3 were unaffected.
- Duration: Approximately 25 minutes (from 21:31 UTC to 21:56 UTC on October 31, 2025).
- Error Seen by Users: Generic “Something went wrong” message when submitting tickets.
Remediation
- Immediate: Add the missing Zendesk configuration in the Prod1 and redployed. Verified successful ticket creation.
- Permanent: Confirmed configuration for all other environments and verified the functionality in each environment
Action Items
- To prevent and catch issues early we are enhancing our automation for zendesk ticket creation flows.