Summary

On July 31, 2026, artifact uploads performed through pipeline in the EU1 cluster began failing with an authentication error. Uploads initiated manually (outside of a pipeline) were not affected, and the ability to retrieve existing artifacts (downloads) was also unaffected — this was isolated to the specific pipeline upload path in one cluster.

Impact

• Artifact uploads performed through pipeline in the EU1 cluster failed with an authentication error for approximately 4 hours and 34 minutes.
• Retrieving existing artifacts (downloads) was not affected.
• Manually uploading artifacts outside of a pipeline was not affected.
• Other clusters/regions were not affected by this issue.

Root Cause

The component responsible for handling pipeline-based artifact uploads is distributed as a container image. In the EU1 cluster, this image is retrieved from an internal registry that mirrors a public image source; in other clusters, the same image is retrieved directly from the public source.

A publishing error in our release process caused a new build of this component to be published using a version label that was already in use, rather than being assigned a new, unique version. As a result, two different images ended up associated with the same version label in the public source.

Our internal registry mirrors images from the public source via an automated replication process. Because of how that replication was triggered, it copied the original (earlier) image associated with that version label rather than the corrected one. This meant the EU1 cluster — which pulls from the internal mirror — ended up running a different, defective image than other clusters, which pull directly from the public source and therefore received the corrected image. The defective image contained an authentication issue that caused pipeline uploads to fail.

Mitigation

• Reverted the affected account to the last known-good version of the upload component, immediately restoring pipeline uploads.
• Published a corrected, permanent version of the component to resolve the issue across all clusters.

Next steps

‌

• Fix the upload step to remove the underlying container-related defect that made this failure mode possible.
• Update our release pipeline for this component so that publishing an image can never overwrite an existing version — every publish must create a new, distinct version going forward.