Issue with In-Progress Pipelines

Summary

Between September 15, 2025 3:30 PM and 4:11 PM PST, customers with actively running pipelines containing service or infrastructure deployment steps experienced execution failures after a Harness platform upgrade. The affected pipelines were unable to resume execution after the upgrade due to a data compatibility issue.

Only pipelines with service/environment steps that were actively running during the upgrade in prod1 and prod2. Pipelines in prod3 and above remained unaffected and no data loss occurred.

Root Cause

During the platform upgrade, an internal data structure change caused active pipelines to be unable to resume execution. The system could not properly process pipeline execution data that was created before the upgrade due to a missing compatibility layer.

Remediation

• Immediate: Customers were advised to restart the failing pipelines, which immediately resolved the issue.
• Permanent: Deployed a compatibility layer to handle data from pipelines that started before platform upgrades, ensuring seamless execution continuity.

Action Items

1. Enhanced Testing – Implement testing procedures to validate running pipeline compatibility during platform upgrades.
2. Improved Change Detection – Strengthen internal monitoring and alerting processes to identify potential compatibility issues during development.