~/brianramsey
Brian Ramsey

brian@ramsey:~$ whoami

Brian Ramsey

I'm Brian, a DevOps Engineer/Manager who is passionate about Technology as well as my hobbies such as AstroPhotograhy, Photograhy, Tiki Drinks and Cooking.

The silent Bottlerocket change that broke our networking

September 12, 2026tags=devops,kubernetes,linux,networking

Upgrading a Kubernetes cluster is usually a routine affair; check check the changelog, then get realistic and check the actual bit of the changelog that really matters. Then check for API Deprecations in the version you are going to, then be lazy and have kubepug actually do that. For us, we also need to intersect the EKS changes as well, because thats how we deploy and run Kubernetes, as well as checking EKS Add-ons version updates, and their corresponding changelog. Satadard, boring stuff right.

That is... right up until a minor version bump forces you to dissect Linux kernel netfilter execution chain. If you are running Amazon EKS, Bottlerocket, and not quite feeling enough pain with your networking configurations, this autopsy of a recent cluster upgrade is for you.

It's worth nothing that none of these issues were encountered with our production workloads; our pre-upgrade testing is conducted separately on a required but not critical cluster.

The Ghost in the Node

Immediately following an EKS control plane and node upgrade from 1.33 to 1.34, workloads began intermittently losing connectivity to ClusterIP Services (most critically, the Kubernetes API server). The failure was entirely non-deterministic per node. Some nodes spun up perfectly healthy; others in the exact same Availability Zone failed identically.

The user-visible impact was immediate: ebs-csi-node and efs-csi-node pods fell into a CrashLoopBackOff, and various controllers failed their leader-election lease renewals with dial tcp <api-server-ip>:443: i/o timeout errors.

The Netfilter Collision

Extensive debugging revealed this was not a transient network blip, but a pure race condition in netfilter hook execution between the AWS VPC CNI and kube-proxy running in nftables mode.

When both of these components target the nftables backend, they install hooks into the kernel's dstnat priority slot (-100):

  • kube-proxy installs a DNAT hook to rewrite the Service ClusterIP to a backend pod IP.
  • AWS VPC CNI installs a CONNMARK hook in the PREROUTING table to mark packets for SNAT.

The Linux netfilter subsystem traverses these hooks on a last-registered, first-executed basis. Registration order on a fresh node is essentially a coin toss.

If the VPC CNI hook executes first (the broken order), the packet (still destined for the non-VPC Service IP) gets marked for SNAT. Routing rules then force the packet out the node's primary ENI. However, because our clusters utilise VPC CNI custom networking, the packet carried a secondary-ENI source IP. AWS EC2 source/destination checks promptly dropped the spoofed packet into the void.

Why Now? The Bottlerocket Shift

The cluster had been running kube-proxy in nftables mode with custom networking for months without a single dropped packet. The catalyst was the jump to the Bottlerocket aws-k8s-1.34 OS variant.

In 1.33, Bottlerocket defaulted to iptables-legacy. The VPC CNI rules lived in legacy xtables, while kube-proxy lived in nftables. Because these are independent kernel data structures, no hook priorities were shared, and no race was possible.

In 1.34, Bottlerocket changed the default iptables backend to nftables, symlinking the host binaries. Suddenly, both the CNI and kube-proxy were writing to the exact same subsystem at the exact same priority.

The DaemonSet Red Herring

During the triage phase, the kube-proxy addon version was bumped. Magically, the broken nodes recovered.

This was a red herring. Rolling the DaemonSet restarted all kube-proxy pods cluster-wide, causing their netfilter hooks to register after the existing VPC CNI hooks. Under the last-registered-first-executed rule, kube-proxy now safely ran first on all existing nodes. Any freshly scaled node, however, was back to a coin toss and continued to fail.

The Workaround

Until upstream fixes landed, the safest mitigation was to force the CNI to ignore your Service CIDR. By injecting AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS=<service_cidr> into the addon configuration, the CNI's connection mark chain returns early for Service traffic before any routing marks can be applied. This neutralises the race outcome regardless of which hook registers first. 

With the release of AWS VPC CNI v1.23.0, the issue has been mitigated upstream, the exact fix was:

Add prerouting connmark rules through nftable instead of iptables. by @yash97 in #3588

Learnings & Warnings

The Bleeding Edge Cuts

Running kube-proxy in nftables mode is an edge case. Even if upstream Kubernetes supports it, the broader ecosystem (like CNIs and host OS networking scripts) often makes implicit assumptions about legacy iptables architectures. Unless you have a rigid organisational requirement or a very deep understanding of the underlying networking stack, stick to standard configurations.

The nuts and bolts of why we even ended up running this configuration is primarily that we were actually running IPVS before that was depreated, and that was being run mainly for the ability to reoute by LC (least connection), having a proper-effective load balancer, that neither iptables nor nftables can actually do. When moving to Bottlerocket we were unable to take IPVS with us, and nftables was chosen as a modern replacement, when we actually should have just reverted back to the default for the service.

R.I.P IPVS 😢

Beware the OS Shim

Kubernetes minor version upgrades are rarely just about API deprecations. Underlying AMIs and OS variants shift silently beneath your workloads. Bottlerocket's backend shim change was the invisible trigger here, turning a stable configuration into a coin-flip outage.

References

no comments yet

←