HostedCI is facing intermittent build failures in few environments

Summary

On Nov 25, 2025 at 6:11 AM IST, build failures began occurring for several customers due to an outage in a third-party dependency (canonical) used during the build initialization process.

Root Cause

During build initialization, an apt-get update step relies on this external dependency. The operation became significantly slower than usual, causing health checks to time out.

Impact

Builds failed during the initialization step between 6:11 AM IST and 6:57 AM IST on Nov 25, 2025.

Mitigation

The third-party outage was resolved externally, and the issue self-recovered once the dependency became available again.

Action Items

• Disable third-party dependency checks during build initialization to avoid similar failures.
• Introduce proactive pre-build checks with fast-fail behavior to detect and mitigate issues earlier.