Few pipelines in prod4 are running with degraded performance

Summary

During a planned security improvement to rotate service authentication secrets in the Prod 4 environment, an incorrect secret value was inadvertently applied to a subset of internal services. This resulted in authentication failures between platform components, impacting CI initialization, delegate task creation, and log streaming for certain pipeline executions. The issue was resolved by rolling back the token configuration to the previous known-good version and validating service recovery.

Impact

Affected environment: Prod 4

Customer-visible impact:

• CI builds failed during initialization with errors such as:

  • “Could not fetch token from log service”
  • “token in request not authorized for receiving tokens” (HTTP 400)

• CD and IaCM pipeline stages continued to execute but did not stream logs.

• Some Shell Script steps failed during execution.

• Delegate tasks could not be created or assigned for some pipeline executions.

No customer data was lost or corrupted.

Root Cause

As part of a planned security initiative, authentication secrets used for communication between internal platform services were rotated in the Prod 4 environment.

During the rotation process, an incorrect secret value was inadvertently configured for one of the services. Instead of applying the newly generated secret in the expected format, a default/incorrect value was introduced. This created a mismatch between services that authenticate using the shared token, causing authentication requests to be rejected with HTTP 400 errors.

The authentication failures prevented CI components from obtaining log service tokens, disrupted delegate task creation, and prevented log streaming for affected pipeline executions.

Resolution

Engineering responded by:

• Reverting the token configuration to the previous known-good secret.
• Restoring consistent authentication across affected services.
• Validating recovery through CI pipeline execution tests.
• Verifying CD and IaCM pipeline execution and log streaming.
• Executing a comprehensive validation suite to confirm platform functionality before closing the incident.

Follow-up Actions

To reduce the likelihood of similar incidents in the future, Harness will:

• Implement additional pre-rotation and post-rotation validation checks to verify that the correct secret values and formats are applied before changes are activated.
• Introduce automated verification of authentication between dependent services immediately following secret rotations.
• Strengthen operational safeguards and deployment validation for security credential rotation procedures to detect configuration mismatches before they can impact customer workloads.