Summary:

Logged in users started getting redirected to the enrollment screen with “Email verified successfully” message and forced users to enter user details again. Pipeline executions and backend tasks were not impacted. Impact was for accounts in Prod 4 cluster.

What was the issue?

We released an incompatible version of Nextgen UI service, resulting in unexpected user flow of new sign up for existing users. This was a human error.

Timeline:

Resolution:

Time Event September 03 7:45 PM UTC Customer reported Login redirection to SignUP page September 03 8:15 PM UTC New deployment happened around the same time. Decided to rollback September 03 8:20 PM UTC Started the partial rollback of FF Proxy changes September 03 8:30 PM UTC Partial rollback didn’t fix the issue. Initiated full rollback September 03 9:00 PM UTC Complete rollback completed and issue resolved

Rollback resolved the issue.

RCA

There was a human error in picking the version of NextGen UI service. Post deployment sanity did not catch this issue. Rolling back took longer than expected as multiple services got deployed together.

Action Item

1. Remove manual process to pick the service versions. Automate the promotion process from lower environments.
2. Improve sanity test to catch above UI flow.
3. Make the rollback process atomic based on the previous known good state.