Pipeline executions failing for only users whose names contain non-ASCII characters

Incident Report for Harness

Postmortem

Summary

On January 26, 2026, a subset of pipeline executions in the production environment failed when triggered by users whose names contained non-ASCII characters. The issue was caused by an encoding error introduced as part of a recent platform change related to propagating user context information between services. The issue was identified, mitigated, and fully resolved through a corrective deployment. 

Impact

During the incident window:

  • Pipeline executions triggered by users with non-ASCII characters in their usernames failed to start.
  • Pipeline executions triggered by users without special characters were not impacted.
  • Pipeline definitions, execution results, and stored artifacts were not affected.

There was no data loss, and the impact was limited to execution initiation for a specific subset of users.

Root Cause

A recent platform enhancement introduced logic to pass user context information between internal services using HTTP headers. This implementation did not correctly encode header values containing non-ASCII characters.

When a pipeline execution was triggered by a user whose name contained special or international characters, the improperly encoded header value caused request validation to fail, resulting in execution failures.

Mitigation

As an immediate mitigation:

  • Customers were advised to temporarily trigger pipelines using a user account without non-ASCII characters in the username.
  • A corrective fix was deployed to properly encode and decode user context data passed between services.

Following these actions, pipeline executions resumed normal operation for all users.

Action Items

To prevent recurrence and improve overall platform robustness, the following actions are being taken:

  • Enforce proper encoding and decoding of all user-provided data used in service-to-service communication.
  • Expand automated test coverage to include users with non-ASCII and international characters in usernames and related user fields.
  • Improve validation across additional user-facing workflows to ensure consistent handling of internationalized input.
  • Introduce broader internationalization-focused test scenarios to detect similar issues earlier in the development lifecycle.
Posted Jan 29, 2026 - 13:53 PST

Resolved

This incident has been resolved.
Posted Jan 26, 2026 - 15:16 PST

Update

We are continuing to work on a fix for this issue.
Posted Jan 26, 2026 - 06:17 PST

Update

All users having no ascii characters failed to execute pipelines.
Posted Jan 26, 2026 - 05:16 PST

Identified

The issue has been identified and a fix is being implemented.
Posted Jan 26, 2026 - 05:10 PST
This incident affected: Prod 1 (Continuous Delivery - Next Generation (CDNG), Continuous Integration Enterprise(CIE) - Self Hosted Runners, Continuous Integration Enterprise(CIE) - Mac Cloud Builds, Continuous Integration Enterprise(CIE) - Windows Cloud Builds, Continuous Integration Enterprise(CIE) - Linux Cloud Builds).