Delegates failing to establish mTLS connectivity due to SSL errors.

At approximately 12:19 PM EST, Aug 26,2025, we had issues where delegates could not establish connectivity over mTLS due to SSL errors. Upon further investigation, we identified that here was an incorrect secret that was rotated causing SSL errors on delegates establishing connection over mTLS. The change was rolled back and mitigated at 1:39 PM EST. 

Root Cause

The issue was traced to an incorrect service secret rotation that occurred earlier in the day around ~ 5:30 AM PDT. This secret rotation was done as part of strengthening our security posture for service-service communication and in preparation for the upcoming release. 

However, incorrect value for this secret led to MTLS configuration load failures in our Global Gateway service which authenticates MTLS traffic and impacted delegate connectivity over MTLS.  The configuration was corrected at ~1:39 PM EST, after which connectivity was fully restored.

Mitigation Actions

Immediate Response

Our engineering team quickly investigated reports and identified the source of the error. The misconfigured secret value was fixed, and services were restored to normal operation.

‌

Next Steps and Improvements

1. Harness is enhancing the internal management on managing secrets and we are adding automated checks to validate secret rotations before and after rollout. So that we can prevent such issues going forward.
2. Alerting will be strengthened to include granular alerts for faster detection of MTLS connectivity issues.