Summary
On December 22, 2025, between 10:55 AM and 2:00 PM CET, customers experienced slowness when accessing and editing pipelines in the Prod3 environment, particularly for pipelines using Harness Code repositories.
Root Cause
The issue was caused by a resource-intensive backend operation that led to elevated system load, resulting in degraded performance for affected workflows.
Impact
- Customer impact: Slowness when accessing and editing pipelines
- Duration: December 22, 2025, from 10:55 AM CET to 2:00 PM CET
- Scope: CODE module and related usage metrics
- No data loss or pipeline execution failures were observed.
Mitigation
The resource-intensive feature was disabled, which immediately restored normal system performance.
Next Steps
- Permanent fix: Optimize backend systems to prevent similar resource contention issues.
- Monitoring: Add enhanced monitoring and alerting to detect early signs of resource saturation.
- Testing improvements: Strengthen pre-production testing to identify and prevent potential resource leaks before deployment.