All Template updates are failing for Prod1

Incident Report for Harness

Postmortem

Summary

Between 19:00 PDT and 21:21 PDT on September 25, 2025, the Template Service experienced an issue due to an incorrect version being deployed. This caused template updates to fail for some customer accounts. The issue was resolved after deploying the correct version, and services have since remained stable.

Root Cause

An incorrect service version was deployed as part of a routine release, which led to compatibility issues in the Template Service.

Impact

Template updates failed for multiple customer accounts during the incident window.

Remediation

Immediate: The correct version was deployed to restore service functionality.

Permanent: Enhancements are being made to the release process to prevent recurrence of similar version-related issues.

Action Items

  • Implement Automated Version Validation: Introduce automated checks to validate deployed versions against the intended release manifest before promotion to production.
Posted Oct 27, 2025 - 03:07 PDT

Resolved

This incident has been resolved.
Posted Sep 25, 2025 - 07:02 PDT

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Sep 25, 2025 - 06:40 PDT

Update

The issue is Fixed now. User should be able to create/update templates now.
Posted Sep 25, 2025 - 06:39 PDT

Update

The issue is Fixed now. User should be able to create/update templates now.
Posted Sep 25, 2025 - 06:38 PDT

Update

We’re aware of the issue and are actively looking into it.
Posted Sep 25, 2025 - 06:36 PDT

Investigating

Any template updates are failing for Prod1.
We are currently Investigating it.
Posted Sep 25, 2025 - 06:30 PDT
This incident affected: Prod 1 (Platform).