Summary

On February 19, 2026, a partial degradation occurred in the CI infrastructure in the us-west1 region due to issues affecting the NAT control plane. During a brief window (~30 minutes), a limited number of CI build jobs failed during VM initialization. The issue was detected through internal monitoring and mitigated via controlled failover, followed by restoration of the affected NAT instances.

Root Cause

‌

The incident was caused by saturation of connection tracking (iptables/conntrack state) on NAT virtual machines in the us-west1 region.

A short-lived spike in build VM activity led to a burst of metadata-related connections. Over time, stale connection entries accumulated without automated cleanup, eventually preventing the NAT VMs from accessing the cloud metadata service.

This metadata connectivity disruption impacted control-plane functionality (including VM provisioning), which resulted in a limited number of build initialization failures.

Impact

‌

• Region Impacted: us-west1

• Customer Impact:

  • Limited CI job failures during VM provisioning
  • Two customers experienced isolated build failures
  • No impact to running workloads

• Data Loss: None

• Duration: Approximately 30 minutes

Customers were advised to retry failed builds after mitigation.

Mitigation

‌

• Traffic was automatically failed over to another NAT to maintain egress functionality.
• Affected s NAT VMs were restarted to clear saturated connection state.
• Metadata connectivity, SSH access, monitoring, and health checks were verified.
• Traffic was gradually restored to affected NAT after stability confirmation.
• Cloud NAT IP utilization was monitored during failover to prevent capacity exhaustion.

Action Items and Permanent Preventive Measures

To prevent such issues from happening again, we will

• Implement automated cleanup of metadata-related connection tracking state.
• Add proactive health checks and alerts for metadata reachability.
• Strengthen monitoring for NAT VM control-plane health degradation.
• Enhance fallback guardrails and capacity validation for Cloud NAT.

• Deploy automated iptables/conntrack cleanup for metadata traffic.

• Improve monitoring for metadata connectivity health.