Harness cloud IDP pipeline failures

Summary
On October 31st, 2025, several IDP pipelines on Harness Cloud infrastructure failed during execution exclusively in the Prod1 environment. The failures were traced to a missing feature flag that had been archived recently while still in use by the IDP service.

Root Cause

A feature flag got archived as part of a routine cleanup. This flag was originally introduced for an internal migration plan. However, the archival validation process only verified references within a limited set of code branches, leading to the incorrect assumption that the flag was no longer in use. When the IDP service attempted to evaluate the archived flag, the evaluation failed, causing pipeline build failures in Prod1 environment.

Mitigation

The issue was promptly mitigated by restoring the archived feature flag, which immediately resolved the pipeline failures.

Action Items

• To prevent such issues and be proactive:

  • We have enhance our sanity checks to ensure no active evaluations exist before flag archival.
  • Expand our archival validation to include all major repositories and release branches.
  • Strengthen the module coverage to ensure all key modules are included in the routine sanity validation process.