Some accounts facing degraded functionalities : PROD1

Impact

Between 05:45 AM and 09:58 AM UTC on Thursday, 24 July 2025, customers using the DevOps Essentials tier in Production 1 encountered “Failed to fetch (500)” errors when attempting to start build pipelines in the UI. All other product areas and tiers remained fully functional.

Root Cause

A recently introduced configuration change created a mismatch between the DevOps Essentials entitlement state and one of the platform’s entitlement-validation endpoints. When that endpoint processed DevOps Essentials requests, it returned an unhandled error path (HTTP 500). Because the UI requires a successful response from this endpoint before running pipelines, build execution was blocked for affected customers.

Timeline (UTC)

Remediation

• Immediate: Rolled back to the last known-good release, restoring UI pipeline functionality for DevOps Essentials customers.
• Permanent: Deployed an updated build with improved entitlement handling logic.

Action Items

1. Strengthen defensive checks – Introduce guardrails that prevent entitlement mismatches from returning unhandled errors.
2. Enhance monitoring & alerting – Add targeted health checks to detect similar discrepancies before they affect customers.
3. Review deployment safeguards – Refine release procedures to ensure configuration changes and validation logic remain in sync.