Pipeline Execution Impacted by Rate Limit

Incident Report for Harness

Postmortem

Overview

We received a report from one of our customers about issues with their pipeline executions in our Prod-2 cluster. Details about the incident are below.

Timeline (PST)

Time Event 7:49 AM One customer reported issue with their pipeline executions 8:45 AM Engineering got engaged in the investigation 9:40 AM Issue identified and mitigated (details below)

Resolution

We deleted the expired Delegate tasks from the database to unblock the customers.

Affected accounts

A total of seven customers were impacted because of this incident. We will take 60 mins of partial downtime for our CD, CDNG, STO and CIE - Self-Hosted Runners.

RCA

We identified the problem with a background job that periodically cleans up expired tasks. The background job hit a latent bug where it was iterating over the same set of tasks and could not delete them. The issue surfaced due to increased database query latency due to a scheduled database upgrade.

We mitigated the incident by manually cleaning up the expired tasks from the database.

Action Items

Enhance our alerting for this scenario so we can catch such issues early.
Improve the background cleanup job to be resilient to db latencies.

Posted Oct 11, 2023 - 17:19 PDT

Resolved

This incident has been resolved.

Posted Oct 11, 2023 - 10:27 PDT

Monitoring

We have cleared the failed/timed-out delegate task queue, and have restored the iterator that processes the queue to operational status and are monitoring the situation closely and will post any additional updates here. All customers should be unblocked at this time.

Posted Oct 11, 2023 - 09:48 PDT

Identified

Harness has identified an issue with an iterator service that cleans up failed and/or timed-out delegate tasks, which is causing the queue to be backed up for some accounts at the threshold of 5000 triggering a rate limit message. Currently, we are clearing the queue of these failed tasks to restore operational capabilities and unblock any affected customers. We have identified the root cause and are working to unblock customers now.

Posted Oct 11, 2023 - 09:35 PDT

Update

We are continuing to investigate this issue.

Posted Oct 11, 2023 - 09:04 PDT

Investigating

Harness Pipelines executions are running into issue with rate limit exceeded. We are currently investigating this issue.

Posted Oct 11, 2023 - 08:08 PDT

This incident affected: Prod 2 (Continuous Delivery - Next Generation (CDNG), Continuous Integration Enterprise(CIE) - Self Hosted Runners).