Incident Summary

Harness FME (Feature Management & Experimentation) faced substantial delays in scheduled impressions data exports on Jul 2, 2026. This issue stemmed from performance degradation within Tinybird's infrastructure, affecting query execution crucial to our export pipeline.

Root Cause

The core issue was a temporary performance degradation on the provider's infrastructure. This directly impacted the metadata retrieval endpoint, which failed to return job IDs intermittently. Consequently, completed export jobs underwent repeated retries, leading to a backlog in the export queue.

Impact

• The export backlog accumulated, peaking at more than 6 hours behind the planned schedule.
• No data loss was detected during the incident.

‌

Mitigation

1. Adjusted Temporal workflows: Extended timeouts from ~30 to ~50 minutes and increased polling attempts from ~180 to ~300.
2. Expanded resource capacity.
3. Traffic redirection: Systematically shifted export workloads over to an optimized in-house cluster.
4. Manual intervention: Terminated stalling workflows stuck in retry loops from failed metadata lookups.

‌

Next Steps

Complete the migration of all remaining export jobs to the internal analytics cluster (In progress).