Impact:
The STO service became unable to process new scan results (pipelines failed) and also display details of previous scans (spinning icon in details panel)
Root Cause:
An expansion of the underlying DB schema was overloading the DB service which was denying cycles to other components, causing timeouts and resulting in failures.
Timeline:
Time
Event
Sep 06, 2023 - 11:06 PDT
DB became unresponsive
Sep 06, 2023 - 11:15 PDT
Issue raised by field
Sep 06, 2023 - 11:28 PDT
Root cause determined to be the DB expansion, became apparent that allowing it to complete would be faster than reverting
Sep 06, 2023 - 11:34 PDT
DB expansion completed. Normal operations continued
Follow-up actions:
- Find an alternate method to update the DB schema.
- Attempt to trigger Pager duty based on successive pipeline failures
- Add monitoring on DB timeout and alert based on the sudden rise