We want to share the details around the deployment failures that impacted our customers [in the Prod-3 cluster] on 12/15 between 4:13 PM — 5:11 PM PT. Users could not trigger the deployments using the Current Gen CD module on https://app3.harness.io.

Root Cause

We do our weekly deployments to app3.harness.io on Thursdays. As part of our deployment, we also add and drop the Mongo database indexes that go along with that deployment. Our Database Reliability Engineering (DBRE) team handles the database indexes.

On 12/15, we had our deployment planned at 4 PM PT, and the DBRE started working on adding and dropping the indexes. Harness, from time to time, regularly looks at ways how we can optimize our Mongo database queries. As part of this effort, on one of the hot collections, a text index was supposed to go in. In Mongo, a collection can have only one text index, and there was an existing text index already present in that collection. So, we had to drop the old text index and create the new one. The syntax of dropping an index is db..dropIndex(). Because of human error and while forming the above statement, the DBRE inadvertently had the statement as db..drop(). This results in Mongo dropping the collection, which happened in our case. 

This collection stores all the CD pipeline execution details. As part of remediation, we recreated the collection and all the indexes, thereby unblocking our customers from triggering all the new workflow executions. We also worked on restoring the historical data into this collection by restoring the data from the database. 

Timeline

3:40 PM PT - DBRE worked on index operations as part of the weekly deployment on app3.harness.io.

3:50 PM PT - Internally, we got alerted that some of the new workflow executions got stuck in the queue state.

4:00 PM PT - 4:20 PM PT - We triggered an internal incident, and during the debugging, the DBRE realized that the Mongo index statement dropped the collection because of the syntax error.

4:20 PM PT - 5:00 PM PT - We started working on parallel things to recover from the incident and restore the data.

• We spun up two new clusters in Mongo, one to restore a Point in Time (PIT) recovery of the database and the other to restore the backup from the snapshot. PIT restore takes a long time, so we took both approaches.
• The engineers on the call realized if we restore the collection with all the indexes, the customers will be unblocked, and the new workflow executions can go through. We also started working on this.
• Snapshot of the data into the new cluster completed around 4:50 PM PT while the PIT cluster restore was still going on.

5:00 PM - 5:11 PM PT - The collection with all the indexes was created, and internally we verified that the new executions were no longer blocked. We also checked with our customers, and they confirmed as well.

5:20 PM - 11:50 PM PT - The next few hours were about restoring the historical data. The PIT cluster restore was completed around 5:23 PM. Using this as the source and our production cluster as the destination, we started importing the data. We also worked on restoring the most current data for our customers and recovered the last two weeks by 6:40 PM PT and the last four weeks by 8:00 PM PT and then continued this activity till all the historic data got restored around 11:50 PM PT.

Remediation 

• This incident can be attributed to a human error. Harness initially had this all automated as part of our deployment for our Current Gen CD, but since this has the problem of taking a long time for a large collection, we have DBREs take care of it. In the current process, before any deployment, there is an automation in place that checks in the code what new indexes got added or removed, compares it with the current database schema, and generates the statements required for the index. The DBREs then review these statements, and if there are any discrepancies, we reach out to Engineering and clarify things. We will look at how to put more controls on the above process. 
• DBREs need the privileges for them to perform day-to-day operations. We are currently exploring creating custom roles for DBREs so that they can only perform specific actions on the database and these inadvertent mistakes don't repeat.

Uptime/Availability

We apply a weightage-based calculation for calculating our uptime/availability https://medium.com/harness-engineering/computing-uptime-for-harness-modules-29be8ae9f622. This incident primarily impacted the deployments and some dashboards for our Current Gen CD customers in Prod-3. This accounts for 40% of weightage.

The users could not trigger new pipeline executions between 4:13 PM — 5:11 PM PT (58 mins/3480 seconds); 40% of it would come to 1392 seconds. So for the week of 12/12 - 12/18, our uptime for our Current Gen CD for the Prod-3 cluster is 99.77.