Intermittent error while loading template service page

Incident Overview

On January 8, 2026, between 10:25 AM and 10:47 AM UTC, the Harness Template Service experienced authentication failures, preventing customers from loading templates.

Impact: Customers were unable to load templates in the Harness UI. No running pipelines were affected.

Root Cause

During a scheduled secret rotation, one database user credential errored out during rotation. This caused the Template Service to lose database authentication, resulting in service failures.

Mitigation Actions

• Immediate Response

  • Re-enabled the old database user to restore Template Service authentication and functionality.

• Resolution

  • Service was fully restored within 2 minutes of detection.

Next Steps and Improvements

• Adding a verification step to confirm the new user is active in GCP Secret Manager before disabling the old user in the database.
• Enhancing the secret rotation process to prevent similar gaps in credential updates