Summary

Dashboards service in Prod3 experienced intermittent failures, causing dashboards to return errors and become unavailable to some customers.

‌

Root Cause Analysis

The incident was caused by slow Looker search_dashboards API calls degrading from 7 seconds to over 30 seconds, which saturated the worker thread pool and prevented the health endpoint from responding to Kubernetes liveness and readiness probes.

‌

Mitigation Steps Taken

Immediate Mitigations

1. Increased liveness probe timeout from 15s to 30s (failure threshold kept at 3, allowing up to 90s tolerance)
2. Doubled thread count , increasing total concurrency
3. Increased pod replica count to maintain higher minimum availability and reduce risk of all pods becoming simultaneously unavailable

‌

Actions

Harness will work on the following Action items to prevent recurrence.

1. Optimize Looker SDK timeout: Tune timeout to Looker SDK search_dashboards calls to prevent indefinite thread occupation
2. Optimize Looker queries : Investigate Looker-side query performance degradation to understand why search_dashboards latency increased