Pipeline failures due to secret decryption in Prod2

Summary:

Pipelines experienced failure in resolving secrets in cases where more than one secret were used in custom secret manager. This issue was isolated to secrets associated with custom secret managers.

Root Cause Analysis:

The pipeline failure happened because the system failed to resolve secrets correctly. A code change to improve performance of the secret decryptions was deployed which resulted in failures for secrets stored in custom secret manager. The code change was behind a feature flag. The feature flag was disabled which restored normal pipeline operations.

Action Items:

1. Add New Test Cases: Add new test cases to the automation suite to cover different configuration combinations for custom secret managers.
2. Add Metrics and Alerts: Implement appropriate metrics and alerts to detect secret/expression resolution failures proactively and mitigate them.