Overview

We experienced a service disruption that affected the pipeline functionality, specifically impacting the pipelines with Test Intelligence steps responsible for uploading call graphs.

Timeline (PST)

Resolution

We have rolled back the recent hotfix to restore the service to its previous state.

Affected accounts

A total of 2 customers and 12 pipelines were impacted by this incident for CIE - Cloud Builds.

RCA

The hotfix aimed to improve the Test Intelligence service by skipping the upload of empty call graphs. However, this change led to the unintended consequence of not serializing empty call graphs that were expected to be present by subsequent steps in the pipeline. The testing for the hotfix was conducted in an environment that did not accurately reflect the production infrastructure where the Test Intelligence service operates. This oversight resulted in the failure to capture the issue before the hotfix was deployed to production.

Action Items

• We are reviewing our testing environments and procedures to ensure they accurately mirror our production settings.
• Thoroughly review and test the change required for skip upload callgraph functionality