Feature Flag SDK authentication operations are running slow in Prod2

Incident Report for Harness

Postmortem

Summary

Between 05:48 PM UTC on March 18 and 04:58 PM UTC on March 19, two customers using app.split.io with SAML authentication using SCIM experienced login failures.

The issue was limited to customers with specific identity provisioning configurations.

Root Cause

A service update was deployed to a login-related component that changed internal request handling behavior.

This caused authentication workflows for SAML-configured customers with identity provisioning enabled to fail during login processing.

Impact

  • 2 customers using SAML authentication with SCIM provisioning enabled were unable to log in to app.split.io.
  • Customers using SAML authentication without SCIM were not affected and could log in normally.
  • Username and password login was unaffected.
  • SDKs, feature flags, and experiments continued functioning normally.
  • No data loss occurred.

Remediation

  • Rolled back the service update that introduced the issue.
  • Restored SAML login for affected customers.

Action Items

To prevent such issues from happening again, we are doing/completed the following:

  • Improve detection capability: Introduced a high-priority alert to detect authentication failures sooner, enabling faster response if similar issues occur in the future.
  • Reduce legacy surface area: Continue guiding remaining customers toward the Harness Platform, where authentication flows are unified and improvements can be delivered more consistently over time.
Posted Mar 30, 2026 - 20:32 PDT

Resolved

This incident has been resolved.
Posted Mar 24, 2026 - 13:37 PDT

Update

We are continuing to monitor for any further issues.
Posted Mar 24, 2026 - 12:03 PDT

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Mar 24, 2026 - 12:03 PDT

Identified

The issue has been identified and a fix is being implemented.
Posted Mar 24, 2026 - 11:25 PDT

Investigating

We are currently investigating this issue.
Posted Mar 24, 2026 - 11:14 PDT
This incident affected: Prod 2 (Feature Flags (FF)).