Possible GCP Networking Issues

Incident Report for Harness

Postmortem

Root cause was GCP outage reported here https://status.cloud.google.com/incidents/skbvoobU4btcxyAthEmu#2c2sBHWU84yPDJ8y1ar4

Posted Oct 24, 2023 - 20:29 PDT

Resolved

While the Google Cloud team continues to work on the mitigation on their end, we will close this incident as resolved since it is isolated to the External Application Load Balancer and won't impact Harness. Thank you.

Posted Sep 22, 2023 - 13:47 PDT

Monitoring

Based on the GCP status page, the issue is isolated to the GCP External Application Load Balancer and the elevated HTTP 4xx Error rates. Harness should not be impacted; we will continue to monitor the situation.

Posted Sep 22, 2023 - 11:29 PDT

Investigating

GCP has posted on their status page an incident affecting Google Cloud Networking, Cloud Load Balancing https://status.cloud.google.com/incidents/skbvoobU4btcxyAthEmu#2c2sBHWU84yPDJ8y1ar4. We validated that app.harness.io has no impact. We will continue to monitor the situation and post updates as they come by.

Posted Sep 22, 2023 - 11:02 PDT

This incident affected: Prod 1 (Continuous Delivery (CD) - FirstGen - EOS, Continuous Delivery - Next Generation (CDNG), Cloud Cost Management (CCM), Continuous Error Tracking (CET), Chaos Engineering, Continuous Integration Enterprise(CIE) - Self Hosted Runners, Continuous Integration Enterprise(CIE) - Mac Cloud Builds, Custom Dashboards, Feature Flags (FF), Security Testing Orchestration (STO), Service Reliability Management (SRM)), Prod 2 (Continuous Delivery (CD) - FirstGen - EOS, Continuous Delivery - Next Generation (CDNG), Cloud Cost Management (CCM), Continuous Error Tracking (CET), Chaos Engineering, Continuous Integration Enterprise(CIE) - Self Hosted Runners, Continuous Integration Enterprise(CIE) - Mac Cloud Builds, Custom Dashboards, Feature Flags (FF), Security Testing Orchestration (STO), Service Reliability Management (SRM)), Prod 3 (Continuous Delivery (CD) - FirstGen - EOS, Continuous Delivery - Next Generation (CDNG), Cloud Cost Management (CCM), Continuous Error Tracking (CET), Continuous Integration Enterprise(CIE) - Self Hosted Runners, Continuous Integration Enterprise(CIE) - Mac Cloud Builds, Custom Dashboards, Feature Flags (FF), Security Testing Orchestration (STO), Service Reliability Management (SRM)), Software Engineering Insights FirstGen (fka Propelo) (Software Engineering Insights FirstGen (fka Propelo) - US), and Service Reliability Management - Error Tracking FirstGen (fka OverOps).