Version 3

InfiniBand for Dummies

Getting Started Tutorial for InfiniBand and Linux

Created by: spiderr, Last modification: 03 Jan 2018 (07:22 UTC) by spiderr
These are mostly my personal notes, culled from various holes on the Internet. I wish there was a basic "Noob's Guide to Networking beyond 1000BaseT (i.e. ye good ol' 1 Gigabit RJ-45 Jack.)

Some of this stuff may not be accurate.. updates appreciated.

Hardware

Mellanox is the king of InfiniBand, though they are selling more Ethernet equipment than InfiniBand these days.

  • Gotcha ConnectX-2 Cards suck because they do not support SR-IOV (which is required for KVM, see below)
  • ConnectX-3 Cards are great!
  • Gotcha ConnectX-3 dual port cards cannot be individually assigned
  • Lowend servers / chipsets (Think Dell R4X0 and below) cannot do SR-IOV may not work properly with the Mellanox cards

Software

  • The Linux kernel has built in InfiniBand support. On CentOS, you can do this: yum groupinstall "Infiniband Support"
  • Gotcha Linux KVM virtualization only supports Ethernet bridging, thus you MUST use SR-IOV if you want InfiniBand in your Guest servers.
Page History
Date/CommentUserIPVersion
03 Jan 2018 (08:34 UTC)
spiderr24.211.144.18211
Current • Source
spiderr24.211.144.1827
View • Compare • Difference • Source
spiderr24.211.144.1825
View • Compare • Difference • Source
spiderr24.211.144.1823
View • Compare • Difference • Source
spiderr24.211.144.1822
View • Compare • Difference • Source
spiderr24.211.144.1821
View • Compare • Difference • Source