Summary

On May 12, 2026, Harness's Secure Connect service experienced a significant disruption that affected customers using Secure Connect for git connector checks and CI pipeline execution. Affected customers experienced connection timeouts and failures when attempting to use Secure Connect-enabled connectors, while connectors operating without Secure Connect continued to function normally.

The disruption was caused by an automated infrastructure maintenance event in our Google Kubernetes Engine (GKE) environment that left a critical load balancer in an unreachable state. The issue was resolved by recreating the affected load balancer and updating the corresponding DNS record to restore connectivity.

Impact:

• Secure Connect-enabled git connector tests failing with connection timeouts
• CI pipelines hanging or failing when Secure Connect was enabled
• Connectors operating without Secure Connect were unaffected

Root Cause

The disruption was triggered by an automated GKE (Google Kubernetes Engine) control plane upgrade on the cluster hosting our internal Secure Connect FRPS (Fast Reverse Proxy Server) infrastructure.

The upgrade executed in two sequential phases in rapid succession. Each phase restarts the GKE cloud controller manager, which is responsible for managing the lifecycle of Kubernetes LoadBalancer service IP addresses. As part of its normal reconciliation process, the controller performs a delete-then-re-insert cycle on IP address reservations when it restarts.

The second phase of the upgrade began before the first reconciliation cycle had fully completed. This created a race condition where the controller deleted the IP address reservation for the internal FRPS load balancer but was interrupted before it could re-acquire it. No Harness application deployments caused or contributed to the issue.

‌

Mitigation

The following steps were taken to restore service:

1. FRPS pod restart — An initial rollout restart of the FRPS deployment was performed. This restored client connectivity to the FRPS server but did not resolve the underlying IP orphaning issue.
2. Load balancer recreation — The internal Kubernetes service (frps-internal) was deleted and recreated, triggering GKE to provision a new load balancer with a new IP address
3. DNS record update — The Cloud DNS A record for was updated to point to the new load balancer IP. This restored end-to-end routing for all Secure Connect traffic.

Full connectivity was confirmed via internal testing and subsequently verified by affected customers.

Preventative Actions

Harness is committed to preventing this class of incident from recurring. The following actions are being implemented: