Helm deployments failing with older delegates. (< 25.05.858XX)

Following a recent release, Helm deployments began failing for customers in the Prod1 and Prod2 environments. The issue primarily affected pipelines running on Delegates with versions ≤ 856xx.

What was the issue?

A serialization incompatibility between the Manager and legacy Delegate versions (version 856xx and below) caused pipeline failures during Helm deployments.

Resolution

Upon identifying the issue, the engineering team immediately rolled back the Manager in both Prod1 and Prod2 to the previous stable version. This action restored pipeline functionality for all impacted customers.

Next Steps

1. The engineering team has already undertaken work to be proactive by adding such serialization checks to catch compatibility issues with older Delegate versions.
2. The team is also adding additional instrumentation and alerting to help surface and detect such deeper issues more effectively.