Degraded functionality in account level templates

Incident Report for Harness

Postmortem

Summary

On November 4, 2025, between 12:00 PM and 3:47 PM IST, customers experienced issues when working with account-level templates in Harness. Affected customers were unable to make changes to their templates via Terraform and encountered pipeline execution failures when using specific account-level templates.

Root Cause

The issue was caused by a database query logic error in the template resolution system. When customers had templates with identical identifiers and version labels at both account and project levels, the system incorrectly returned project-level template when account-level templates were requested. This was due to missing exact scope filtering in the database query logic.

Impact

  • Scope: Account-level template operations and pipeline executions using affected templates
  • Duration: 3 hours and 47 minutes
  • Data Loss: None - all template data remained intact

Remediation

  • Immediate: Workarounds were suggested to customers during the investigation phase to change version labels / identifiers for project level templates so that we break this identifier matching combination. 
  • Permanent: Deployed a fix that corrected the database query to properly filter for account-level templates with appropriate scope filtering

Action Items

  1. To prevent and catch such issues going forward, we have automated tests to cover template resolution scenarios with overlapping identifiers across different scopes, same to be done across other entities as well
  2. We are also eeviewing similar database queries across the codebase to identify and fix potential scope filtering issues
Posted Nov 14, 2025 - 09:05 PST

Resolved

This incident has been resolved.
Posted Nov 04, 2025 - 03:09 PST

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Nov 04, 2025 - 02:29 PST

Investigating

We are currently investigating this issue.
Posted Nov 03, 2025 - 23:53 PST
This incident affected: Prod 2 (Continuous Delivery - Next Generation (CDNG)).