Prod2: Users are unable to access Harness UI

Incident Report for Harness

Postmortem

Impact:

Harness UI was inaccessible. All pipelines currently under execution and triggers were operational.

‌

Root Cause:

In the 0.353.11 UI release, we introduced a feature flag to control login flow for certain customers. Due to a misconfiguration on our feature flag side, these customers were impacted and their UI screen was not loading

‌

Detailed Timeline:

Time Event July 11, 2023 18:15 PST Few customers reported that UI was not loading fine. July 11, 2023 18:30 PST Harness identified that this was because of a recent UI release. July 11, 2023 18:31 PST UI release was rolled back which resolved the issue. Customers were notified

‌

Remediation Plan and Action Items:‌

What was done to remediate the current Issue?

‌ UI release was rolled back to 0.353.10.

‌

Why didn’t we roll back early?

This issue was not reproducible in our QA and various prod accounts. We initially suspected it to be caused by backend account level configuration since all our internal production accounts were not affected.

‌

How can this be prevented in the future?
- We identified a bug in UI logic to consume the response of the feature flag response. This is fixed now.
- We identified a gap in automation for P0 and P1 issues with UI feature flag metrics. This is actively being worked upon to rectify.
- Additionally, we have set up segment metrics that would help monitor and proactively alert in such scenarios to mitigate such issues even more promptly.

Posted Jul 12, 2023 - 17:11 PDT

Resolved

We can confirm normal operation. Get Ship Done!
We will continue to continue to monitor to ensure stability, additionally, we are investigating the root cause and will publish a postmortem soon.

Posted Jul 11, 2023 - 18:45 PDT

Monitoring

Service issues have been addressed and normal operations has been resumed. We are monitoring the service to ensure normal performance continues.

Posted Jul 11, 2023 - 18:31 PDT

Investigating

The Harness UI is currently inaccessible to users. We are currently working to identify the root cause and restore the service as soon as possible.

Posted Jul 11, 2023 - 18:15 PDT

This incident affected: Prod 2 (Continuous Delivery (CD) - FirstGen - EOS, Continuous Delivery - Next Generation (CDNG), Cloud Cost Management (CCM), Continuous Error Tracking (CET), Continuous Integration Enterprise(CIE) - Mac Cloud Builds, Feature Flags (FF), Security Testing Orchestration (STO), Service Reliability Management (SRM)).