Seeing intermittent login issues on our Prod4 environment

Summary:

Users experienced login failures on the Prod4 cluster due to backend connection limits being exceeded. The issue was triggered by a surge of WebSocket connections following a customer account migration, which triggered the circuit breaker limit on Harness Global Gateway for the Prod4 cluster.

Timeline:

Resolution:

• Increased backend capacity by scaling the Global Gateway service to distribute the load more effectively.
• Set up necessary alerts to monitor system stability to confirm cluster connectivity.

RCA:

Following a customer account migration, there was a significant increase in WebSocket connections from delegate agents, exceeding the connection limits set for backend hosts. The backend system reached its maximum capacity, preventing new connections from being established. Additionally, one of the backend pods restarted unexpectedly, leaving only a single pod to handle all incoming traffic. This led to the circuit breaker being activated, causing login failures.

‌

Action Item:

• Implement a dedicated traffic splitting configuration to handle WebSocket connections separately from other API requests to prevent similar incidents in the future.
• Improve monitoring and alerting to detect when connection limits are approaching critical thresholds.
• Conduct scalability testing to ensure the system can handle large numbers of WebSocket connections without reaching critical limits.