Code services are degraded, git clone is failing

Incident Report for Harness

Postmortem

Summary

Between June 19 and July 17, 2026, the built-in Git Clone step — and any pipeline step using the drone-git clone plugin — failed on ARM64 Kubernetes build infrastructure with the error exec /usr/local/bin/clone: exec format error. AMD64 (Intel/AMD) builds, Windows builds, and the VM containerless binary path were not affected.

The root cause was a defect in our internal image publishing process that caused ARM64-tagged drone-git images to actually contain AMD64 binaries. We identified and mitigated the issue the same day it was reported by reverting the drone-git image to the last known-good version. No customer action or configuration change was required.

Root Cause

On June 19, 2026, a security remediation restructured how the drone-git image is built. AMD64 builds were updated correctly, but the ARM64 build pipeline didn't build ARM64 files directly — it adapted the AMD64 build file via text substitution and compiled it on ARM64 infrastructure. The June 19 change altered the AMD64 file so that substitution silently no-op'd instead of failing, so the pipeline published an image tagged ARM64 whose Git Clone and Git LFS binaries were still compiled for AMD64.

Impact

  • Affected: The built-in Git Clone step, and any pipeline step using the drone-git clone plugin, running on ARM64 Kubernetes build infrastructure, across all accounts, between July 6 and July 17, 2026.
  • Symptom: Builds failed at the Git Clone step with exec /usr/local/bin/clone: exec format error.
  • Not affected: AMD64 (Intel/AMD) Kubernetes and VM builds, Windows builds, the VM containerless execution path, and our hardened image variant.

Mitigation

We reverted the drone-git image version used across all affected services to the last known-good release. This fully resolved the ARM64 execution failures; no customer configuration changes were required.

Next Steps

To prevent such issues from happening again.

  • Rebuild the ARM64 image publishing pipeline to build our dedicated ARM64 build files directly, rather than adapting the AMD64 build files.
  • Enhance automated post-publish validation to every image release: verify binary architecture matches the image tag, and run a functional smoke test before an image is considered releasable.
  • Expand automated test coverage to include ARM64 Kubernetes build scenarios.
  • Remove the affected intermediate image versions from circulation once the corrected release is validated.
Posted Jul 22, 2026 - 10:13 PDT

Resolved

This incident has been resolved.
Posted Jul 16, 2026 - 00:01 PDT

Investigating

We are currently investigating this issue.
Posted Jul 15, 2026 - 23:40 PDT
This incident affected: Prod Eu 1 (Code Repository).