Impact:

The impact was only in Prod 2 cluster

• The Pipelines failed to load and execute
• Fetch Pipelines was working fine.
• Pipeline YAML validations were failing for users.

‌

Root Cause:

The cause was determined where the schema generated on the Prod 2 cluster was incomplete. We encountered an issue with the dynamic schema creation process, where multiple requests for the same account were being handled concurrently within the same thread pool. As a result, there were potential stitching issues with the schema generation.

‌

Detailed Timeline:

Time Event June 8th 1:09 PM IST First customer reached out with failure to save and execute pipelines, internal incident call was triggered for investigation. June 8th 1:22 PM IST Pipeline team identified the issue is due to an incomplete schema. June 8th 1:23 PM IST Schema cache is invalidated and to safe guard we enabled an FF that will ensure no impact to customers due to schema cache. June 8th 1:29 PM IST All our sanity checks started working in the customer account and we closed the incident call after confirmation in slack. June 8th 1:35 PM IST We notified in the internal incident channel that issue was resolved.

‌

Remediation Plan and Action Items:

Short Term Solution

1. We disabled the schema validation for customers on prod1 and prod2. This will not have any adverse impact on the customer as their executions will still proceed. Schema validations are providing errors using JSON schema, which using Harness Yaml Editor will still be valid.

Long Term Solution

1. We are currently in the process of deploying the static schema to the production environment. This schema, which is already live in the pre-production environment, does not rely on dynamic creation. Instead, it is released as a predefined and validated configuration. By ensuring that the schema is thoroughly validated during the sanity checks, we can guarantee that any incorrect schemas will be identified and resolved prior to deployment.
   Consequently, the schemas available in the production environment will always be valid. This feature implementation provides us with the assurance that incidents caused by invalid schemas in the production environment will be effectively prevented.

2. We will be using Prometheus Metrics to expose custom metrics in order to proactively capture similar issues:

   1. To capture the number of failed executions due to schema validation per customer
   2. Number of successful executions per account