Users are not able to create new environments or target groups in the Feature Flag Module

Incident Report for Harness

Postmortem

Overview:

On Monday, September 18th, an issue was detected, impacting the ability to create new feature flag environments or target groups.‌

The incident occurred following release to one of the UI modules that the feature flag UI uses to control user access.

Timeline (PDT)

Time	Event
2:00 AM	Issue detected by internal testing
2:18 AM	Confirmation that the issue replicated in Prod1 and Prod2
2:50 AM	Confirmed issue related to a recent release by the SSCA module that the Feature Flags UI uses
3:00 AM	Rolled back SSCA release in QA, and confirmed issue was no longer present
3:30 AM	SSCA release rolled back in Prod1 and Prod2
3:43 AM	Monitoring
4:19 AM	The issue has been confirmed as resolved

Resolution

Once identified, the change to the module was rolled back, and functionality was restored.

Affected Users

Feature flag admin users in Prod1 and Prod2

RCA

A number of Feature Flag UI tests failed. Upon investigation it was discovered that users were unable to create new environments, or create TargetGroups via the UI. When attempting to create these, the user was shown an error message.

Upon inspecting the source of the error in the developer console, the source of the error was identified to be coming from the SSCA JS package that was remotely loading on Modal popup. When this remote JS package was blocked, users where able to create the TargetGroup.

Reviewing recent releases, a release for the SSCA module aligned with the failed tests timeline.

Action Items

Align Feature Flag testing around dependent UI module releases in testing environments
Broader testing for each UI module prior to release, to incorporate services that rely on the modules

Posted Oct 24, 2023 - 01:46 PDT

Resolved

We can confirm normal operation. Get Ship Done!
We will continue to monitor and ensure stability.

Posted Sep 18, 2023 - 04:19 PDT

Monitoring

Harness Feature Flag with creating environment or target groups issues have been addressed and normal operations have been resumed. We are monitoring the service to ensure normal performance continues.

Posted Sep 18, 2023 - 03:43 PDT

Investigating

Users are not able to create new environments or target groups in the Feature Flag Module. We are working to identify the cause and restore normal operations as soon as possible.

Posted Sep 18, 2023 - 02:18 PDT

This incident affected: Prod 1 (Feature Flags (FF)) and Prod 2 (Feature Flags (FF)).