Impact:

In Prod1 environment, the tests page was not responsive on some CI pipelines. The page loads intermittently.

Root Cause:

The TI API handlers talk to TimeScale DB to fetch the test summary data which in turn hook into a distributed tracing framework. The framework SDK calls were taking longer than expected resulting in the Test Report APIs (upload and fetch) timing out.

Timeline:

11:54 AM PDT Internal FireHydrant got created.

12:08 PM PDT Deployed the TI service with tracing disabled.

Downtime: 15 mins after FireHydrant was triggered. Partial (only CI pipelines with test reports configured)

Action Items:

• Remove tracing for TI service which has been known to cause instability.
• Enhance logging
• Add pprof API endpoints to the TI service to debug issues in QA and pre-QA.