Impact:

STO-core api service calls made from the UI intermittently threw 403 Forbidden responses, even though user was authorized

Root Cause:

STO Core’s authorization codepath was modified in such a way that a check for mismatches between a JWT’s sub claim and the request’s Account ID was performed for user requests. Previously this check had only be done for cross-service tokens. The issue was not detected in QA because it was a result of the interaction between multiple services that was not covered by automated tests. The issue was related to a bug in another service on an edge case and was intermittent.

Timeline:

Remediation:

For immediate resolution, we rolled back the deployment. Rollback took longer than desired because of a version mismatch with a database migration happening concurrently with the deployment

Action Items:

1. Remediate causal bug in JWT handling
2. Decrease alerting thresholds on 403s within STO-core
3. Improve automation on DB rollback