Scheduled maintenance for Prod1 to isolate Redis instances

Scheduled Maintenance Report for Harness

Postmortem

Maintenance Summary: We have successfully completed the critical maintenance required to isolate the Redis instances for our Prod1 cluster. This update is pivotal for enhancing our system architecture.

Details:

Start Time: 08:00 AM PT Jul 20, 2024
End Time: 09:55 AM PT Jul 20, 2024
Impact: Affected in-flight pipelines in Prod1, with no impact on Prod2.

Benefits Achieved:

Localized Fault Zones: Enhancements in fault isolation for increased system robustness.
Performance Improvements: Enhanced efficiency and response times.
Better Debugging Capabilities: Streamlined and more efficient troubleshooting processes.

Incident Impact: During the maintenance, in-flight pipelines were impacted. Necessary actions were taken by aborting stuck pipelines, and all related cleanup activities are now complete.

Resolution and Recovery: All services have been restored to full functionality. We are monitoring the systems closely to ensure stability.

Next Steps: Customers experiencing any issues post-maintenance are encouraged to contact Harness support for assistance.

We appreciate your understanding and support as we continue to enhance our services.

Posted Jul 23, 2024 - 06:06 PDT

Completed

The scheduled maintenance has been completed.

Posted Jul 20, 2024 - 09:00 PDT

In progress

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Posted Jul 20, 2024 - 08:00 PDT

Update

We will be undergoing scheduled maintenance during this time.

Posted Jul 19, 2024 - 08:38 PDT

Scheduled

We are committed to providing a reliable and secure environment for your data. In light of the recent incident, we will be performing a critical maintenance activity on our Prod1 Harness cluster. This maintenance is essential to isolate our Redis instances, ensuring enhanced security and performance stability.

Maintenance Details:

Date: July 20th, 2024
Time: 8:00 AM to 9:00 AM Pacific Time
Impact:

During this window, pipelines in-flight and new executions started in Prod1 Harness cluster will get impacted.

Other services including the Prod2 Harness cluster will remain operational.

Post-maintenance, the Prod1 and Prod2 Harness clusters will be completely isolated, preventing any cross-interference and enhancing our operational integrity.

We apologize for any inconvenience this may cause and appreciate your understanding as we enhance our infrastructure.

For any immediate concerns, please contact our support team at support@harness.io.

Thank you for your patience and cooperation.

Posted Jun 17, 2024 - 10:34 PDT

This scheduled maintenance affected: Prod 1 (Continuous Delivery - Next Generation (CDNG), Continuous Integration Enterprise(CIE) - Self Hosted Runners, Continuous Integration Enterprise(CIE) - Mac Cloud Builds, Security Testing Orchestration (STO)).