Editing 'Variable Sets' in the IaCM module is experiencing issue

Incident Report for Harness

Postmortem

Executive Summary

On August 4, 2026, between approximately 3:36 PM and 9:00 PM IST, customers using Infrastructure as Code Management (IaCM) on Prod0 and Prod1 were unable to access the Variable Sets settings page. The page rendered blank with no error message, and customers with Variable Sets attached to their workspaces could not view or manage them for the duration of the incident. Prod2, Prod3, and EU1 were not affected.

Separately, during the same window, a scheduled maintenance action caused the IaCM settings tab to temporarily disappear across all environments. This was identified and reversed within the incident bridge call before significant customer impact occurred.

We deployed a hotfix that restored full access to the Variable Sets page on Prod0 and Prod1 the same evening, and we are implementing permanent safeguards described below to prevent this class of issue from recurring.

Impact

  • Customers with the Variable Sets feature enabled on Prod0 and Prod1 were unable to view or manage Variable Sets for approximately 5–6 hours.
  • No data was lost or corrupted, this was a UI routing failure only; underlying Variable Sets data and configuration were not affected.
  • Prod2, Prod3, and EU1 were not affected by this issue.
  • A secondary issue, a scheduled feature flag operation caused the IaCM settings tab to temporarily disappear across all environments during the incident bridge call. This was identified and reversed within minutes. External customer exposure for this secondary issue is still being confirmed.

Root Cause

A platform routing change released on July 18, 2026 updated how IaCM settings pages are resolved in the user interface. As part of that change, any settings page that had not been explicitly re-registered in the new routing structure became unreachable.

The Variable Sets page had not been re-registered under the new routing structure, making it inaccessible in the environments where the routing change had been deployed — Prod0 and Prod1. Because the failure occurred at the routing layer rather than within the page itself, the page rendered blank with no visible error rather than showing a clear failure message.

Remediation

Immediate

We deployed a hotfix that re-registered the Variable Sets page in the updated routing structure, restoring access for all affected customers on Prod0 and Prod1.

Permanent

We are adding automated end-to-end tests that navigate to settings pages with relevant feature flags enabled, configured as a required gate in our release pipeline. We are also documenting and enforcing the routing constraint through static analysis so that settings pages are never inadvertently left out of the routing structure during future platform changes.

Action Items

To prevent such issues from happening again,

  1. Enhance automated tests, that navigate to settings pages with relevant feature flags enabled, configured as a blocking gate in the release pipeline, so this class of regression is caught before it reaches production.
  2. Establish an explicit checklist step for future platform-wide architectural changes that verifies all existing settings pages remain accessible in the updated routing structure before the change is promoted to production.
Posted Aug 06, 2026 - 10:07 PDT

Resolved

This incident has been resolved.
Posted Aug 04, 2026 - 08:28 PDT

Update

We are continuously rolling out fixes to all Clusters, Prod3 has been resolved.
Posted Aug 04, 2026 - 06:16 PDT

Update

We are continuously rolling out fixes to all Clusters, Prod EU1 has been resolved.
Posted Aug 04, 2026 - 06:09 PDT

Update

We have identified the issue and started to implement the fix , prod2 is restored.
Posted Aug 04, 2026 - 06:00 PDT

Update

We are continuing to investigate this issue.
Posted Aug 04, 2026 - 05:17 PDT

Update

We are continuing to investigate this issue.
Posted Aug 04, 2026 - 05:08 PDT

Investigating

We are currently investigating this issue.
Posted Aug 04, 2026 - 05:07 PDT
This incident affected: Prod Eu 1 (Infrastructure as Code Management (IaCM)), Prod 1 (Infrastructure as Code Management (IaCM)), Prod 3 (Infrastructure as Code Management (IaCM)), and Prod 2 (Infrastructure as Code Management (IaCM)).