Summary

On March 17, between approximately 3:04 PM and 4:05 PM PST, customers experienced degraded performance and intermittent unavailability for Test Intelligence (TI) related APIs. This impacted test result uploads and report access.

Root Cause

A recent change introduced additional background processing during test result ingestion, which significantly increased database load. This led to resource saturation and caused elevated latency and temporary service disruption.

Impact

• Degraded performance for test uploads and report retrieval
• Intermittent API unavailability (~7 minutes)
• Affected customers experienced slower response times across TI-related workflows 

Remediation

Immediate

• Increased database capacity to stabilize performance
• Applied configuration changes to reduce system load

Permanent

• Optimized processing logic to reduce database overhead
• Introduced safeguards to prevent similar load amplification scenarios

Action Items

• Improve performance validation using production-scale datasets
• Enhance safeguards for new feature rollouts
• Strengthen monitoring to detect abnormal load patterns earlier
• Add controls to limit impact scope of similar changes