Certain users are unable to see feature flags in prod1 and prod2

Incident Report for Harness

Postmortem

Summary

On July 14, 2026, certain Harness Feature Flag (FF) Classic customers on the Prod1 and Prod2 production environments were unable to view Feature Flags in the Harness platform. Affected requests returned an authorization error (HTTP 403), so Feature Flags were not visible for those users until access was restored.

The behaviour was caused by a planned security update that began requiring an additional Feature Flags permission for related read operations. Users whose roles did not yet include that permission were correctly denied access, which appeared as a product failure. Harness temporarily rolled back the Feature Flags service change to restore access, updated the required permissions for affected users and roles, and confirmed that Feature Flag visibility returned to normal. The stronger permission checks remain in place going forward.

Impact

During the customer-reported incident window on July 14, 2026 (status page approximately 11:25 UTC to 11:57 UTC):

  • Certain Feature Flag Classic customers on Prod1 and Prod2 were impacted.
  • Affected users could not view Feature Flags in the Harness UI / API and received 403 Forbidden responses.
  • Impact was limited to users and roles that did not yet have the updated Feature Flags permission required by the security change.
  • A public status page update was posted for Prod1 and Prod2 Feature Flags.

There was no data loss, no change to stored feature flag configurations, and no impact to Feature Flag evaluation for applications whose SDK credentials and permissions were unaffected. Customers and users with the required permission continued to operate normally.

Root Cause

Harness deployed a planned Feature Flags authorization update that enforces the ff_targetgroup_view permission on target-related read paths used when viewing Feature Flags. This enforcement is intentional and remains the expected behaviour.

Users and roles that had not yet been granted ff_targetgroup_view received 403 responses and could not see Feature Flags. From the customer’s perspective this looked like an outage; it was an authorization denial due to missing required permissions after the security update.

Mitigation

Harness completed the following mitigation steps:

  • Temporarily rolled back the Feature Flags service in Prod2 and Prod1 (and aligned Prod0) to restore access quickly while permissions were corrected.
  • Updated / granted the required ff_targetgroup_view permission for affected users and roles.
  • Verified that Feature Flag visibility returned to normal and closed the status page incident.

These actions restored customer access. The permission enforcement introduced by the security update remains in effect going forward; the lasting fix is correct permission assignment, not removal of the check.

Preventive Actions

To avoid a recurrence of this issue, Harness is taking the following actions:

  • Keeping the stronger Feature Flags permission checks in place as the permanent security posture.
  • Ensuring role and permission updates for ff_targetgroup_view are applied with (or before) similar authorization changes in production.
Posted Jul 23, 2026 - 15:32 PDT

Resolved

This incident has been resolved.
Posted Jul 14, 2026 - 04:57 PDT

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Jul 14, 2026 - 04:56 PDT

Investigating

Certain users are unable to see feature flags in prod1 and prod2
Posted Jul 14, 2026 - 04:25 PDT
This incident affected: Prod 1 (Feature Flags (FF)) and Prod 2 (Feature Flags (FF)).