Prod3: Slowness in Harness Platform

Summary

Between July 7 and July 8, 2026, customer accounts hosted in our Prod 3 environment experienced intermittent slowness and failures when loading pipelines and resolving templates. The underlying cause was a sharp, sustained increase in internal feature-flag lookup traffic from our template-processing service to our core platform backend service.

The issue presented across two separate days. On Day 1 (July 7), our team restored service through infrastructure level mitigations by restarting affected services and adding capacity. Because that underlying cause was still present, the same failure mode recurred on Day 2 (July 8). Deeper investigation on Day 2 identified the specific feature flag and code path responsible, and engineering shipped a fix so that flag check is now served from local cache. This fix was deployed across all production clusters and fully resolved the issue; it has not recurred since

We are auditing all other feature flags on the platform for the same caching gap, and are implementing process and monitoring improvements described in the Preventive Measures section below to catch this class of issue earlier and reduce the risk of recurrence.

Impact

Symptoms: Intermittent slowness or failure loading pipelines; slow or failed template validation and resolution; intermittent connectivity/timeout errors between platform services.

Functional impact: Pipeline executions and pipeline-studio operations that depend on template resolution were delayed or failed during the active windows described in the timeline above. Some executions required manual retry

Root cause

Our core backend service could not keep up with the request arrival rate because of a sustained, uncached feature-flag lookup driving a large and growing volume of internal traffic following a recent release.

‌

Mitigation

• Immediate mitigation (Day 1 and Day 2): removed the saturated backend instance(s) from service and added capacity to relieve the acute bottleneck.
• Permanent fix (Day 2): updated the template-processing service so the responsible feature-flag check reads from local cache instead of calling the backend service on every lookup, eliminating the excess internal traffic at its source.
• The fix was deployed to all production clusters and verified through a return to normal internal traffic volumes and response times.

Preventive Measures & Next Steps

• Feature-flag caching audit: We are reviewing all feature flags used in high-frequency code paths across the platform to identify and close any similar caching gaps before they can cause a repeat of this issue.
• Release validation: We are strengthening pre-release load testing for new feature-flag-gated code paths so that call-volume regressions of this kind are caught before reaching production.
• Capacity alerting: We are adding proactive alerting on backend request-queue saturation so that this class of bottleneck is caught and mitigated automatically, before it affects customer-facing response times.