Prod1: Users are unable to access Harness NextGen UI

Incident Report for Harness

Postmortem

A microservice that handles access control was deployed to our prod-3 cluster, but the namespace was incorrectly configured to that of our prod-1 cluster. These are logical clusters in the same physical kubernetes cluster, so the prod-3 configuration overwrote the prod-1 values, causing the outage in prod-1. As a result, users could not access Harness UI and api calls were failing with Http status 403 in the prod-1 cluster.

The input variables were selected manually and deployed, rather than using the preconfigured input sets that have the correct sets of values. To prevent such human errors going forward we are putting pipeline policies in place to validate the inputs have correct environment and namespace combinations.

Posted May 08, 2023 - 12:31 PDT

Resolved

We can confirm normal operation. Get Ship Done!
We will continue to continue to monitor to ensure stability, additionally, we are investigating the root cause and will publish a postmortem soon.

Posted May 08, 2023 - 02:52 PDT

Monitoring

Service issues have been addressed and normal operations has been resumed. We are monitoring the service to ensure normal performance continues.

Posted May 08, 2023 - 02:17 PDT

Investigating

The Harness NextGen UI is currently inaccessible to users. We are currently working to identify the root cause and restore the service as soon as possible.

Posted May 08, 2023 - 00:58 PDT

This incident affected: Prod 1 (Continuous Delivery - Next Generation (CDNG), Cloud Cost Management (CCM), Continuous Integration Enterprise(CIE) - Mac Cloud Builds, Feature Flags (FF), Security Testing Orchestration (STO), Service Reliability Management (SRM)).