This server's network bond was set to layer2 hashing, which picks which NIC to use based on MAC address. Since there are only two endpoints — our server and the switch — this meant nearly all incoming traffic ended up on one NIC and all outgoing on the other. Not balanced at all.

At normal load this didn't matter. But when Ceph storage traffic spiked (probably a scrub job), the receiving NIC hit its 10G ceiling. The NIC's buffer overflowed — over 10 million packets dropped — and incoming network packets failed CRC checksum validation. No data on disk was affected.

Here's where it got bad: when libceph (the kernel storage client) sees a CRC failure, it tears down the connection and retries everything in-flight. Every retry means the storage servers re-read and resend the data, which adds more traffic to the link that's already full, which causes more CRC failures, which triggers more retries. A feedback loop across all 182 storage connections and 78 virtual disks on this server.

The real kicker: libceph has no retry limit and no timeout — this is a known issue in the kernel client. The storm literally cannot stop on its own.

This node has 256G RAM and 128 vCPUs — it's used for CPU-heavy VMs that need less RAM, which is why it only has 2x10G networking instead of the 25G or 100G links on our bigger nodes. At normal load that's plenty, but the combination of krbd + imbalanced bonding + smaller pipes made it the one node where this could happen. All our other nodes were already on layer3+4 hashing and librbd. This setting had been wrong since day one.

1. Throttled the link with tc to 2 Gbit — broke the retry loop, CRC errors stopped immediately
2. Fixed the bond hash from layer2 to layer3+4 so traffic gets spread across both NICs
3. Moved VMs to other nodes
4. Rebooted VMs that went read-only
5. Rebooted the node to clear stale Ceph connections