Impact

Users were unable to log in to app.propelo.ai and app.levelops.io through the https://app.propelo.ai/signin flow.

Data ingestion, data processing, propels and any user already logged in in the US region were unaffected.

The EMEA and APAC regions were not affected.

Workaround: The users would still be able to log into the system via SEI flow as this issue impacts only the /signin flow of the application

‌

Root Cause

One of the login flows is affected by tenant deletion tasks as the login flow looks at all the tenants in order to find out which tenants the user trying to log in has access to.

• If the users table of the available tenants doesn't exist, the flow fails entirely.
• During tenant deletions, several tenants were removed but the entry for the tenant to be considered 'available' was not deleted.
• The connection to the DB was severed due to a VPN disconnect.

‌

Timeline

Time Event 2023-10-02 05:45 AM PDT The issue was resolved 2023-10-02 05:40 AM PDT The tenants which were marked for deletion were removed from the global list of tenants and operations are normal post the deletion of these unused tenant ids. 2023-10-02 05:10 AM PDT Issue was identified via internal testing and incident was triggered.

‌

Action Items

• Institute a downtime window and alerting mechanism to stakeholders on the maintenance activity.

  • Also, perform verification / run sanity tests across tenants in respective regions to ensure the app is up and running.

• Review the logic for /login and add any guardrails w.r.t the check on the global tenants list.

• Institute a process/tool for tenant de-provision on the legacy SEI module.