Harness NextGen CD pipelines are failing with Webhook based triggers

Incident Report for Harness

Resolved

We can confirm normal operation. Get Ship Done!
We will continue to continue to monitor and ensure stability.
Posted Feb 25, 2022 - 09:47 PST

Update

Fix have been deployed to all affected clusters, and we are continuing to monitor for any issues.
Posted Feb 25, 2022 - 09:09 PST

Monitoring

Harness service issues have been addressed in Prod2 and normal operations have been resumed. We are monitoring the service to ensure normal performance continues. We are deploying the same fix post verification to Prod1 now and will provide another update once the deployment completes.
Posted Feb 25, 2022 - 07:45 PST

Update

We have identified the cause of the issue and are working on a fix. Please continue to monitor this page for updates.
Posted Feb 25, 2022 - 05:39 PST

Identified

We have identified a potential cause of Harness NextGen CD pipelines failing with Webhook based triggers and are working hard to address it. Please continue to monitor this page for updates.
Posted Feb 25, 2022 - 04:08 PST
This incident affected: Prod 1 (Continuous Delivery (CD) - FirstGen - EOS) and Prod 2 (Continuous Delivery (CD) - FirstGen - EOS).