Summary
At 1:47am GMT, automated synthetic checks simulating SDK traffic for the Feature Flags service began failing,
Subsequent investigation revealed that one of the FF service dependencies was partially degraded due to a missing container image in Google Artifact Registry (GAR).
Root cause
During a routine deployment, Kubernetes rescheduled some pods onto new nodes that did not have the image cached locally. Since the image had been deleted from GAR, some of the new pods failed to start,
Mitigation
The image was restored, the impacted pods were redeployed, and normal functionality resumed.
Impact
- Some requests to http://config.ff.harness.io/client/auth received 502 responses.
- New SDKs starting up during the incident may have encountered 502 errors when attempting to authenticate with Harness SaaS.
- SDKs include a built-in retry mechanism with exponential backoff for 5xx responses, so clients using up-to-date SDKs should have eventually re-authenticated successfully.
- Existing SDKs that had already authenticated with Harness Saas won’t have experienced this issue.
Preventive Measures
- Review and strengthen image retention policies in Google Artifact Registry to prevent unintended deletions.
- Implement automated validation to ensure that all production images are retained for a defined minimum period.
- Add monitoring and alerting for image pull errors/