BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage News How Google Traced GKE API Timeouts to a Linux Kernel TCP Bug

How Google Traced GKE API Timeouts to a Linux Kernel TCP Bug

Listen to this article -  0:00

Google recently detailed how three interacting bugs across Google Kubernetes Engine (GKE), Private Service Connect, and the Linux kernel caused intermittent Kubernetes API connectivity failures, blocking around 800 developers from deploying to production.

In January, a customer running multiple GKE clusters with Private Service Connect (PSC) for private control-plane access experienced intermittent timeouts when pods attempted to reach the Kubernetes API server.

Packet captures revealed that the control plane was responding to new TCP connection requests with stale ACK packets belonging to previously closed connections. The root cause involved three separate bugs across different networking layers. Akash Thakur, senior technical solutions engineer at Google Cloud, describes how engineers across multiple Google teams identified three root causes and ultimately addressed them:

In a race to fix the problem, we involved engineers across Google Kubernetes Engine (GKE), Private Service Connect, networking, hostnet, kernel, and the Linux upstream community, who together tracked down not one but three distinct root causes. And the final fix landed not as a config change or a feature flag, but as a patch to the Linux kernel itself.

The investigation identified three interacting causes: packet reordering in PSC, Envoy's half-close behavior preventing TCP connections from being properly closed, and a Linux kernel bug that discarded certain reordered FIN packets.

Source: Google Cloud blog

Thakur describes a complex troubleshooting case involving I/O timeouts across Private Service Connect, virtual network interfaces, and the Linux TCP state machine, which ultimately revealed a kernel issue leaving sockets stuck indefinitely in the FIN_WAIT_2 state. Thakur explains:

Most networking cases resolve at one layer. You find the misconfigured firewall rule, the exhausted connection pool, the miscalculated MTU. You fix it and move on. This one was different because the symptom — i/o timeout on a connection that never completed — was the same whether you were looking at the Private Service Connect layer, the kernel's TCP state machine, or Envoy's connection management. Every layer had a plausible explanation, and every explanation was incomplete.

The issue was addressed through three coordinated fixes over several weeks: Google first rolled back a Private Service Connect change to restore connectivity, then corrected the proxy protocol handling and added regression tests. Finally, a Linux kernel patch, authored by Eric Dumazet, formerly a software engineer at Google, fixed the underlying TCP issue. Rohan Karne, managing director at Google Cloud, comments:

This is the unglamorous reality of mission-critical platform engineering—methodically ruling out the obvious suspects and digging into packet captures until you find the kernel root cause.

Thakur highlights how individually reasonable behaviors across networking, proxies and operating systems can combine into complex infrastructure failures. Packet captures at multiple network layers were essential to identifying the problem, while correlating network symptoms with infrastructure changes helped establish the connection between the three bugs. Guangba Yu, a postdoctoral researcher at the Chinese University of Hong Kong, highlights:

For AIOps and chaos engineering, the transferable method is organizing simultaneous path observations, protocol state, and rollout events while distinguishing symptom recovery from state-machine correction. This anonymized case supplies no population failure rate and does not explain all API timeouts.

Justin Garrison, field CTO at Sidero, notes:

At first I was excited to read this article that leads to a kernel patch which are always fun. It turns out GKE just runs a kernel from 3 years ago (6.1) and the bug was already fixed upstream and they had to backport it.

The kernel fix was merged into Linux 6.9.7 and backported to the 6.6 and 6.1 LTS branches, reaching GKE customers in version 1.33.5 and later.

About the Author

Rate this Article

Adoption
Style

BT