Summary

On February 16, 2026, at 7:04 PM UTC , an incident occurred that caused pipeline execution failures for workflows using cross-scope secrets. Affected customers were unable to execute pipelines that referenced secrets stored in a different scope than the pipeline’s execution scope.‌ The issue was introduced as part of a recent production changes. The system was rolled back to the previous stable release to restore normal functionality.‌

Root Cause

A regression introduced in a recent release impacted the secret resolution flow for cross-scope secret managers.‌

Under specific conditions, the system incorrectly resolved the scope of the secret manager during decryption, causing the lookup to fail. As a result, pipelines referencing cross-scope secrets were unable to complete secret evaluation successfully.‌

The issue was not detected prior to production deployment due to a gap in automated test coverage for cross-scope decryption scenarios.

Impact

Customers using cross-scope secrets experienced pipeline execution failures.

Pipelines that relied on secrets stored in a different scope than the pipeline execution scope were unable to run successfully during the impact window.

Other pipeline executions that did not use cross-scope secret managers were unaffected.

‌

Mitigation‌

Immediate

‌

• Rolled back the production deployment to the previous stable release.
• Functionality was fully restored following the rollback.

Permanent

• Enhanced test coverage to include cross-scope secret evaluation scenarios.
• Updated automation to validate pipeline execution using non-default Secret Managers.
• Reviewed and improved alerting for error rate monitoring related to bulk encryption and secret resolution flows.
• Reviewed deployment rollback processes to prevent similar environment inconsistencies.
• Strengthened code review validation for scope-sensitive changes.

Action Items

To prevent such items in the future

• Add comprehensive automation coverage for cross-scope secret decryption workflows.
• Enhance and further solidify CI/CD validation for non-default Secret Manager scenarios.
• Improve monitoring and alerting for secret resolution error rates.