Harness SEI Service Unavailable

Incident Report for Harness

Postmortem

Impact

Some of the customer's dashboards in the US region were not loading (partially or completely) intermittently.

‌

Root Cause

The SEI operations were degraded for a few hours due to a problem with error handling of a database query in one of the services. The system went back to normal after restarting the service. A fix was deployed to correct the bug and prevent the issue from happening again.

‌

Timeline

Time Event 2023-08-16 2:49 pm PST Hotfix deployed 2023-08-16 8:48 am PST Service Restart 2023-08-16 7:21 am PST Customer Escalation 2023-08-16 5:04 am PST DB Error Logging Spike

‌

Remediation Plan and Action Items:

What was done?

Fix the error handling in the code path with the defect - HF Released

How can this be prevented in the future?

Review other transactions and fix the error handling if necessary.
Improve alerting and monitoring to catch these types of failures earlier.

Posted Aug 20, 2023 - 00:41 PDT

Resolved

We can confirm normal operation. Get Ship Done!
We will continue to monitor and ensure stability.

Posted Aug 16, 2023 - 09:15 PDT

Monitoring

Service issues have been addressed and normal operations has been resumed. We are monitoring the service to ensure normal performance continues.

Posted Aug 16, 2023 - 08:48 PDT

Identified

We have identified a potential issue causing the service access problem and are working hard to address it. Please continue to monitor this page for updates.

Posted Aug 16, 2023 - 07:21 PDT

Investigating

The Harness SEI operations are facing performance degradation. We are currently working to identify the root cause and restore the service as soon as possible.

Posted Aug 16, 2023 - 05:04 PDT

This incident affected: Software Engineering Insights FirstGen (fka Propelo) (Software Engineering Insights FirstGen (fka Propelo) - US).