On April 16, 2026, Hosted CI Linux pipelines experienced intermittent initialization failures due to an upstream outage affecting package repositories used during environment setup.

Impact

A subset of customers running Hosted CI pipelines encountered failures during the initialization phase, preventing jobs from starting successfully.

• Affected: 24 accounts
• Failed executions: 129 pipelines
• Impact duration: ~5 hours 49 minutes

Root Cause

The issue was caused by a service disruption in an external package repository provider. During CI environment provisioning, dependency installation requests to this upstream service timed out, causing initialization failures.

Remediation

Immediate Mitigation
We updated runner configurations to bypass dependency installation during initialization and rolled out updated environments across affected clusters.

Permanent Fix
We have improved resilience in our CI infrastructure by:

• Using pre-configured environments with required dependencies pre-installed
• Eliminating runtime dependency on external package repositories during initialization
• Enhancing failure handling for external dependency timeouts

Action Items / Next Steps

• Continue improving isolation from external dependencies during environment startup
• Strengthen monitoring and alerting for upstream service degradation
• Optimize rollout speed for infrastructure changes to reduce mitigation time