Cross-Flashing a Mellanox ConnectX-3 Pro: Or How I Learned to Stop Worrying and Love InfiniBand
I needed 40 Gbps between two machines: intel9: Ubuntu 24.04, running K8s + misc services xeon2zfs: TrueNAS SCALE 25.10.3, serving storage via NFS The plan: Deploy InfiniBand FDR (40 Gbps), run IPoIB for network services, eventually layer NFS over RDMA if I felt fancy. The hardware: Two Mellanox ConnectX-3 Pro cards from eBay ($20 each), one unmanaged QDR/FDR switch ($35), some cheap QSFP cables. Should be plug-and-play, right? VPI cards, support both Ethernet and InfiniBand, auto-negotiate, just works™. ...