CI degradation with CI steps using AWS connector with inherited authentication

Incident Report for Harness

Postmortem

Summary

On April 2, 2026, customers experienced failures in CI pipelines during S3 upload steps following a routine delegate upgrade. The issue primarily impacted customers using cross-account AWS role assumption with inherit-from-delegate connectors.

Impact

Few customers across Prod1 and Prod2 using CI pipelines using S3 upload with cross-account role assumption experienced Artifact uploads failures, blocking downstream deployments

Root Cause

A change introduced during the delegate upgrade altered how AWS credentials were passed to CI steps. This resulted in partial credentials being provided to the S3 upload plugin, which triggered a latent issue in the plugin’s credential selection logic.

Instead of executing the intended cross-account role assumption flow, the plugin attempted authentication using incomplete credentials, leading to failures. 

Mitigation

  • Rolled back delegate to the previous stable version
  • Restored original credential handling behavior
  • Service functionality recovered immediately after rollback 

Next Steps

To prevent such issues from happening again we will:

  • Improve validation of credential handling in CI steps
  • Expand automated test coverage for cross-account scenarios
  • Reintroduce changes behind proper feature flags with full end-to-end testing
Posted Apr 17, 2026 - 08:30 PDT

Resolved

This incident has been resolved.
Posted Apr 02, 2026 - 10:53 PDT

Investigating

We are investigating a degradation in CI steps when using AWS connectors and inherited authentication.
Posted Apr 02, 2026 - 08:13 PDT
This incident affected: Prod 2 (Continuous Delivery - Next Generation (CDNG)) and Prod 1 (Continuous Delivery - Next Generation (CDNG)).