Template service is experiencing issues when marking templates as stable.

Incident Report for Harness

Postmortem

Summary

On Nov 10th at 22:58UTC, Users were unable to save templates as a stable version. This issue affected a minor workflow. The failure occurred because the system attempted to construct a scopeInfo object with a null uniqueId when a feature flag (FF) was disabled. Since the template save path requires a valid uniqueId, the operation failed. Customers were advised to avoid this flow until a fix was deployed. A mitigation was applied the same day, and a permanent fix was later deployed to production.

Impact

  • Template save operations that attempted to mark a template as “stable” failed.
  • This affected a secondary, non-critical flow and in all environments where the Feature Flag was disabled while the new scopeInfo logic was active.

Root Cause

The failure occurred because scopeInfo was constructed with a null uniqueId when the relevant feature flag was disabled. A recent change introduced strict validation for uniqueId, which caused the template save operation to fail when provided with a null value.

Action Items

  1. To prevent such issues happening we are enhancing our test coverage and feature flag coverage by:

    1. Adding validation at API boundaries to ensure uniqueId is always present or handled safely.
    2. Expand automated test coverage to include FF-on and FF-off scenarios for template save flows.
    3. Add guardrails and structured logging to detect null uniqueId cases earlier.
Posted Nov 30, 2025 - 22:13 PST

Resolved

This incident has been resolved.
Posted Nov 11, 2025 - 07:56 PST

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Nov 11, 2025 - 07:50 PST

Identified

The issue has been identified and a fix is being implemented.
Posted Nov 10, 2025 - 22:04 PST

Monitoring

The issue has been identified and a fix is being implemented.
Posted Nov 10, 2025 - 22:03 PST

Identified

We have identified that the template service is experiencing issues where customers are unable to save templates as stable versions. However, they can still use the templates without marking them as stable, which serves as a temporary workaround.
Posted Nov 10, 2025 - 14:08 PST

Investigating

We are currently investigating an issue in pipeline templates while promoting the version to stable.
Posted Nov 10, 2025 - 14:07 PST
This incident affected: Prod 2 (Continuous Delivery - Next Generation (CDNG)) and Prod 1 (Continuous Delivery - Next Generation (CDNG)).