Infiniband

🎬 ABERTURA Câmera abre num galpão escuro. Luzes piscando. Som de cooler. Ao fundo, uma silhueta masculina com 24 cores se forma na fumaça. É ULTRA2. Plim plim. Entra a música tema: 🎵 “Detalhes tão pequenos de nóóóós… Um swap que ninguém pediu, um kernel que não rodóóóu…” 🎵 CAPÍTULO 1: A CHEGADA DO FORASTEIRO Era uma noite quente em Debene do Sul, pequena fazenda de servidores às margens do Rio MoCA 2.5. Dona Felipe, matriarca da família e dona de todas as terras (e dos /24 inteiros, que ela mesma fez questão de subnetar com as próprias mãos), recebia um forasteiro vindo de longe. ...

🎬 OPENING CREDITS Camera opens on a dark warehouse. Blinking lights. Sound of cooling fans. In the background, a masculine silhouette with 24 cores forms in the mist. It’s ULTRA2. Plim plim. (That’s the iconic Brazilian soap opera sound effect.) Theme song plays: 🎵 “Such tiny details of us two… A swap nobody asked for, a kernel that wouldn’t boot through…” 🎵 CHAPTER 1: THE STRANGER ARRIVES It was a hot night in Debene Ranch, a small server farm by the MoCA 2.5 River. Dona Felipe, matriarch of the family and owner of all the land (and all the /24 subnets, which she personally configured by hand), received a stranger from afar. ...

I needed 40 Gbps between two machines: intel9: Ubuntu 24.04, running K8s + misc services xeon2zfs: TrueNAS SCALE 25.10.3, serving storage via NFS The plan: Deploy InfiniBand FDR (40 Gbps), run IPoIB for network services, eventually layer NFS over RDMA if I felt fancy. The hardware: Two Mellanox ConnectX-3 Pro cards from eBay ($20 each), one unmanaged QDR/FDR switch ($35), some cheap QSFP cables. Should be plug-and-play, right? VPI cards, support both Ethernet and InfiniBand, auto-negotiate, just works™. ...

There’s a moment in every homelab evolution where you stop and ask: what am I actually buying when I buy infrastructure? For me that moment came two weeks ago, staring at a renewal email from a SaaS I won’t name, doing back-of-the-envelope math on a Synology I almost ordered, and pricing out an EKS cluster for a side project that — let’s be honest — was never going to leave my apartment. ...