A daemon for deprecated security configurations from a third-party vendor in our production Google Cloud infrastructure has been a primary contributing cause to the issue. The daemon has incorrectly added IP table entries inside the nodes that were running the Feature Flag services, causing packet drops.
Jan 4, 23:43 PST - We got some alerts around load increase in our Feature Flag services, and customers also reported Feature Flag SDK’s returning 502 responses.
Jan 4, 23:43 PST - Jan 5, 01:00 PST - We started debugging the issue, and we checked the dashboards for load, memory, and CPU-related spikes, as well as the logs, and we noticed 502 responses showing up in the logs. We redeployed the Feature Flag services to see whether this would bring the situation back to normal.
Jan 5, 01:00 PST - Jan 5, 03:00 PST - We engaged with our Google support as we noticed our Network Load Balancer had unhealthy backends for one of the Auth services within the Feature Flag module. We changed the configs and redeployed the auth service, which didn't resolve the problem. We did a local port forward to the Feature Flag Services bypassing the load balancer, and the service health showed green. So the debugging focus shifted more towards figuring out what's happening within the Network load balancer as we see the traffic from Load Balancer not reaching the Backend.
Jan 5, 03:00 PST - Jan 5, 05:00 PST - With the Google support on-call, we continued to focus our debugging by SSHing into the nodes and pods where the Feature Flag services were running and captured the network packet captures using tcpdump. The analysis of these packets showed us that the packets were reaching the node but could not reach the pods.
Jan 5, 05:00 PST - Jan 5, 07:00 PST - While the GCP support continued to look into their internal logs to understand the behavior, we redeployed the Feature Flag services with some additional changes to gauge any further insights that we could get from the logs.
Jan 5, 07:00 PST - Jan 5, 09:00 PST - We created a new Network Load balancer pointing to the same backends and tried to see whether this would address the issue, but we still saw 502 responses getting returned. We still saw high retransmission errors from the nodes running the Feature Flag services, so we again captured the network packets using tcpdump to check whether we could gather any new debugging information.
Jan 5, 09:00 PST - Jan 5, 11:00 PST - We continued to work with the GCP to isolate the issue; we deployed new node pools and moved the services over to these node pools but without much success.
Jan 5, 11:00 PST - Jan 5, 14:00 PST - We moved some of the Feature Flag service components to our fail-over cluster in a different GCP region. This addressed the problem partially, wherein all the existing Feature Flag flag evaluations continue to work but not the creation of the new flags.
Jan 5, 14:00 PST - Jan 5, 16:20 PST - GCP support continued to debug on their end, and during this time, we started delving deep into the underlying networking of Kubernetes along with the GCP support. Upon SSHing into the nodes and looking at the IP Table rules, we found that the deprecated security configurations from a third-party vendor were blocking and rejecting the traffic from reaching the pod. The daemon has incorrectly added IP table entries inside the nodes that were running the Feature Flag services, causing packet drops. We deleted the third-party daemon set across all the nodes, drained the nodes, and launched the nodes of the Feature Flag services on the new node pool. This resolved the issue on Jan 5th at 16:20 PST, and the functionality was restored across all the environments.
In our ongoing effort to continuously improve and strengthen our security posture, we have implemented layered technologies to detect and prevent malicious activity in our cloud and corporate environments. As part of the transition to new solutions, we had legacy security monitoring and enforcement tooling running in Production to facilitate tuning and a graceful transition to the new tools. That legacy tooling detected what it thought was a malicious payload originating from our Load Balancer (i.e, inbound traffic external to the Harness). It took programmatic action to block the suspected malicious traffic. Since the LB IP addresses were not part of the "safe IP address" list, a "block" protection policy was matched against the HTTP request, resulting in the LB IP address being programmatically blocked.
After confirming that the target payload was not malicious and was a false positive alert generated by the legacy tooling, we have now completely removed the third-party daemon set that caused this issue. To ensure no remnants of the IP table rules existed across the entire fleet of Harness services, we drained all the old nodes from our Production GKE cluster, made sure that the new nodes got spun up, and all the services got deployed onto these new nodes.
We have also engaged with the third-party vendor to understand further why these signals triggered this incident, as this software has been running without any issues in our Production environment for the last 20 months.
A component of our layered strategy for security monitoring is to minimize tooling that requires “in-line” deployment to inspect for malicious activity. While the legacy solution that caused this issue did require this style of deployment, the current set of tools we have deployed does not.