A subset of CI jobs may fail during the initial clone step in K8 environment

Incident Report for Harness

Postmortem

Summary

‌

A new version of the git clone plugin was deployed to enable enhanced analytic events within CI pipelines. This update included a change to the plugin’s entrypoint. Customers who had pinned older versions of the clone plugin were still using images that relied on the previous entrypoint, leading to failures in the clone codebase step for Kubernetes-based CI pipelines.

Additional Context:
While the primary impact was on customers using pinned older plugin versions, some executions using the latest plugin version were also affected during rolling deployment and rollback. In these cases, different steps of the same execution were routed to CI-Service pods running different versions, causing inconsistent behavior.‌

Root Cause Analysis (RCA)

The issue occurred due to a version mismatch between the updated CI system and older, pinned versions of the git clone plugin.
The updated CI-Service expected the new entrypoint introduced in the latest plugin. However, older pinned plugin images still referenced the previous entrypoint, which caused the following error during execution:

fork/exec /usr/local/bin/drone-git: no such file or directory

Since pinned plugin versions differ across customers, only a subset of users was initially affected. Once the plugin version alignment issue was identified, affected pipelines were restored by adjusting CI service configurations to ensure compatibility.

Additional Context:
During rolling deployment and rollback, some executions—regardless of plugin version—were split across CI-Service pods running different versions. For these executions, the init step ran on one pod version while the git clone step ran on another, leading to failures even for customers using the latest plugin.

Action Items / Preventive Measures

  1. Broader Plugin Compatibility Testing
    We are expanding our testing to include commonly pinned plugin versions, ensuring compatibility issues are identified before release.
  2. Improved Version Handling
    We will introduce safeguards in CI-Service to better handle differences between plugin versions, preventing similar mismatches from causing pipeline failures.
  3. Deployment Safety Enhancements
    We will review and implement mechanisms to ensure all steps of a pipeline execution run on the same CI-Service pod version during rolling deployments or rollbacks.
Posted Nov 25, 2025 - 17:44 PST

Resolved

This incident has been resolved.
Posted Nov 19, 2025 - 14:01 PST

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Nov 19, 2025 - 13:46 PST

Identified

Impact: customers using older version of drone-git image (k8s infra) pinned version are impacted because entry point is changed for the image.
Posted Nov 19, 2025 - 13:34 PST

Investigating

We are currently investigating this issue.
Posted Nov 19, 2025 - 13:34 PST
This incident affected: Prod 1 (Continuous Integration Enterprise(CIE) - Linux Cloud Builds) and Prod 2 (Continuous Integration Enterprise(CIE) - Linux Cloud Builds).