Harness CI Cloud builds for Linux-Arm64 are impacted

Summary

A recent Docker update (v29) introduced changes that caused compatibility issues between the Docker Moby client and the Docker Engine. As a result, Linux ARM builds were impacted, leading to failures during CI pipeline initialization. The issue was isolated to environments that automatically pulled the latest Docker version.

To restore stability, the Docker CE version has been pinned to v28.x, ensuring compatibility with the existing Moby client and preventing further build disruptions.

Root Cause Analysis (RCA)

The incident occurred because the Docker Moby client used in our build infrastructure was not fully compatible with the newly released Docker Engine (v29). This mismatch caused errors during container initialization, preventing CI pipelines from starting. Since our setup scripts were designed to install the latest available Docker version by default, the update propagated automatically, resulting in failures across Linux ARM-based build environments.

By reverting to a known stable version (Docker CE v28.x), the incompatibility was resolved, and all pipelines resumed normal operation.

Action Items / Preventive Measures

To prevent recurrence of similar issues, the following measures have been implemented and planned:

• Version Pinning: Docker CE has been pinned to v28.x to ensure consistent and stable builds across all environments.
• Image Baked Dependencies: Going forward, Docker and other critical dependencies will be baked directly into the base image, eliminating the need for dynamic installation during runtime and avoiding exposure to unvalidated upstream updates.