Users using the Harness CI Test Intelligence or Test Report management observing pipeline failures

Incident Report for Harness

Postmortem

Impact

Prod-1 Customers using Test Intelligence or Test Reports management feature with CI pipelines from 7:48 AM to 8:08 AM PST noticed their pipelines failing.

‌

Mitigation

Restarted the prod1 ti-service pods.
Doubled the ti-service prod1 memory to 24Gi.
Added profiler to figure out the root cause.

‌

Timeline

Time Event 7:03 AM Sept 28 2023 Engineering got alerted about Test Intelligence Service restarts. 7:45 AM Sept 28 2023 Internal FireHydrant Incident created to check degraded performance of Test Intelligence service. Test report uploads taking longer than usual. 7:48 AM Sept 28 2023 Test reports upload failing due to Test Intelligence service being down. More restarts. 8:08 AM Sept 28 2023 Crash loop is stopped and engineering is monitoring the pod resource consumption. Test Intelligence service function becomes normal. Proactive measures to increase pod memory and replica counts started. 11:45 AM Sept 28 2023 Memory profiler support added for further debugging. 3:00 PM Sept 28 2023 Root cause determined and additional action items noted.

‌

Root Cause

Why did some CI steps fail to upload test reports?

A Harness internal service called Test Intelligence service which is responsible for intelligent test selection and also test report management went into a crash loop.

Why was the Test Intelligence service crashing?

A significant increase in memory usage was identified, leading to the Test Intelligence service entering a crash loop. As a consequence, requests related to Test Selection and Test Reports management experienced failures, disrupting the CI pipeline steps reliant on these functionalities.

What caused the significant increase in memory usage?

Harness recently added Test Intelligence support for pipelines triggered by git Push, but it unintentionally skipped deleting old database entries, causing an Out-of-memory issue and service restarts. This affected CI pipelines using Test Intelligence and those reliant on test reports.

‌‌

Follow-up action Items:

Took proactive steps to mitigate the issue by increasing resources assigned to the service and cleaning up stale database entries to reduce any further impact.
Fixed the issue causing the entries to keep accumulating and deployed it.
We are working on optimizing database queries to only fetch required fields from each entry to reduce memory consumption
Add alerts for proactively monitoring database sizes

Posted Oct 04, 2023 - 20:27 PDT

Resolved

We can confirm normal operation. Get Ship Done!
We will continue to monitor and ensure stability.

Posted Sep 28, 2023 - 08:40 PDT

Monitoring

Harness test intelligence service issues have been addressed and normal operations have been resumed. We are monitoring the service to ensure normal performance continues.

Posted Sep 28, 2023 - 08:08 PDT

Identified

We have identified a potential cause as a large memory spike observed due to which the ti-service is getting OOM and crashing. We are working hard to address it. Please continue to monitor this page for updates.

Posted Sep 28, 2023 - 08:00 PDT

Investigating

Users using the Harness CI Test Intelligence or Test Report management observing pipeline failures. We are working to identify the cause and restore normal operations as soon as possible.

Posted Sep 28, 2023 - 07:48 PDT

This incident affected: Prod 1 (Continuous Integration Enterprise(CIE) - Self Hosted Runners, Continuous Integration Enterprise(CIE) - Mac Cloud Builds).