Few users facing issue while loading or creating Chaos Experiments

Incident Report for Harness

Postmortem

Impact:

Harness Chaos Engineering module page was resulting in errors for a few users and they were unable to view or create Chaos Experiments.

‌

Root Cause:

During a restart of one of the chaos manager pod it encountered an issue as it could not find the SEGMENT_API_KEY secret. This resulted in a CreateContainerConfigError.

‌

Detailed Timeline:

Time Event June 5, 2023, 02:35 PST One of the pod in the replica set got restarted. July 5, 2023, 03:12 PST Internal testing found the issue. July 5, 2023, 03:36 PST Fixed the Redis secret key and resolved the issue

‌

Remediation Plan and Action Items:‌

What was done to remediate the current Issue?

‌ The secret was updated with the correct key.

‌

Why didn’t we roll back early?

‌ It was a manual error in updating by overwriting a key instead of adding a new key.

‌

How can this be prevented in the future?

We have changed the pipeline to inherit the values from the secret manager.
Added a process to bounce the related pods if dependent config maps or secrets are updated.

Posted Aug 01, 2023 - 07:40 PDT

Resolved

We can confirm normal operation. Get Ship Done!
We will continue to monitor and ensure stability.

Posted Jun 05, 2023 - 04:10 PDT

Monitoring

Harness Chaos Engineering service issues have been addressed and normal operations have been resumed. We are monitoring the service to ensure normal performance continues.

Posted Jun 05, 2023 - 03:36 PDT

Identified

We have identified a potential cause of the service issues and are working hard to address it. Please continue to monitor this page for updates.

Posted Jun 05, 2023 - 03:12 PDT

Investigating

The Harness Chaos Engineering service is experiencing performance issues. We are working to identify the cause and restore normal operations as soon as possible.

Posted Jun 05, 2023 - 02:35 PDT

This incident affected: Prod 2 (Chaos Engineering).