What was the issue?

After a recent change to the Feature Flag authentication gateway, some evaluations failed for TargetGroups with rules that use a custom attribute.

Once identified, the team reverted the configuration change, and evaluations returned to normal

RCA

As part of improvements to our disaster recovery strategy, a change was made to make the Feature Flag authentication gateway more robust.

Initial testing preformed failed to account for the scenario of client-side SDKs, with target groups using rules that use a custom attribute (rather than a core attribute like identifier).

Client SDKs generally make two types of request.

1. An auth request
2. An evaluation request to get flag values.

During the auth flow, the provided target and its attributes is stored in a DB and Redis e.g.

{
.  identifer : "123-456-789",
.  name : "bob",
.  custom_attribute_1 : "value1"
}

During the evaluation flow, the target is retrieved from the cache if still present, and if not, it will be retrieved from the DB, and stored in the cache.

After the Feature Flag authentication gateway change, targets were being written to a different Redis. They would still be persisted to the DB, but if the Redis instance used during evaluations contained an older version of the target that did not have the attributes, then the code would never go to the DB i.e. the Redis may contain

{
.  identifer : "123-456-789",
.  name : "bob"
}

In this case, if the custom rule used identifier or name, it would work as expected, but if it is using the custom attribute, then that would be missing during the evaluation.

Actions Items

1. Update our test suite, to include additional user authentication flows, to account for the impacted use case
2. Update the Feature Flag authentication gateway to use distinct Redis instances for each environment
3. Review functionality, that supports additional reading of target attributes, to provide a failsafe to ensure the correct evaluation is returned