Summary

CI runners in the us-west1 and us-central1 regions intermittently experienced connection timeouts of approximately 134 seconds when reaching external services such as GitHub and Bitbucket over outbound network gateways.

Impact

• CI runners in the affected regions intermittently experienced connection timeouts of approximately 134 seconds when reaching external services (e.g., GitHub, Bitbucket) over our outbound network gateways.
• The issue was intermittent rather than constant — connections succeeded under normal load, and failures clustered during periods of high outbound traffic volume.
• No data was lost or corrupted. This was a network-connectivity and capacity issue, not a data-integrity issue.
• us-west1 and us-central1 were the affected regions; other regions were not impacted by this issue.

Root Cause

‌

Our load balancer distributes outbound traffic across multiple NAT gateways using a hashing method based on connection details (source/destination address and port). For any single connection, these details stay constant for that connection's lifetime. We had a sustainted traffic surge for a few seconds which congested the gateways

‌

Action Items

To prevent such issues from happening again Harness will,

Increase outbound connection capacity on our NAT gateways by provisioning additional external network interfaces, giving each gateway a substantially larger pool of connections it can serve concurrently..