Prod3: Unable to Create Environments,Service Overrides and Infrastructure

Incident Report for Harness

Postmortem

Summary

Users in the Prod3 cluster were unable to create entities such as Environments,Service Overrides and Infrastructure.

Resolution

A fix was implemented to address the system issue. As a result, the system stabilized, enabling users to successfully create Environments and other affected entities.

Timeline

Time (UTC) Event
March 4, 2025, 8:35 AM UTC Issue reported for unable to create Environments,Service Overrides and Infrastructure
March 4, 2025, 11:34 AM UTC A fix has been implemented and we are monitoring the results.
March 4, 2025, 11:59 AM UTC Incident resolved

The Root Cause Analysis (RCA)

Harness was previously having an issue with login in the Prod3 cluster. To address that, Harness had reverted the system release to the previous version. This rollback led to data inconsistencies, resulting in failures for certain entities, such as Environment and Service overrides. The inconsistencies were later resolved in a subsequent update, after which functionality was confirmed to be working as expected.

Posted Mar 11, 2025 - 13:12 PDT

Resolved

This incident has been resolved.
Posted Mar 04, 2025 - 03:59 PST

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Mar 04, 2025 - 03:34 PST

Identified

The issue has been identified and a fix is being implemented.
Posted Mar 04, 2025 - 00:35 PST
This incident affected: Prod 3 (Continuous Delivery - Next Generation (CDNG)).