InfiniBand vs Ethernet Performance

Blowing past 1GbE in your data center, Baremetal vs KVM virtio, NFS rdma vs default 40GbIB vs 40GbE vs 10GbE with SR-IOV and RDMA

Created by: spiderr, Last modification: 05 Jan 2018 (14:59 UTC)
This is not intended to be a comprehensive "real world" test, however it is an apples to apples test using an NFS mount and a simple dd speed test to compare InfiniBand and Ethernet in a virtualized and non-virtualized environment.

NFS Export

[root@guest1 ~]# cat /etc/exports
[root@baremetal2 ~]# nfsstat -m
/mnt/56ib from
 Flags:    rw,relatime,vers=4.1,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=,local_lock=none,addr=

Infiniband vs Ethernet Performance Stats, Results are all Per Second
Connection40Gb IB Mellanox/EMC SX600540Gb Eth Cisco NX306410Gb Eth Cisco NX3064
16K * 64KB16K * 64KB16K * 64KB
Server Client Write Read w/Cache Write Read w/Cache Write Read w/Cache
Baremetal Baremetal 3.2Gb/s 409MB/s 9.8Gb/s 1227MB/s 47.8Gb/s 5984MB/s 5.6Gb/s 700MB/s 8.69Gb/s 1086MB/s 49.4Gb/s 6184MB/s
KVM SR-IOV Baremetal 4.2Gb/s 529MB/s 9.6Gb/s 1206MB/s 47.2Gb/s 5910MB/s 5.5Gb/s 689MB/s 9.7Gb/s 1216MB/s 44.8Gb/s 5602MB/s 5.0Gb/s 631MB/s 8.7Gb/s 1088MB/s 46.4Gb/s 5802MB/s
KVM SR-IOV KVM SR-IOV 4.5Gb/s 659MB/s 7.1Gb/s 893MB/s 40.1Gb/s 5014MB/s 6.5Gb/s 816MB/s 7.0Gb/s 884MB/s 45.3Gb/s 5669MB/s
KVM Bridge KVM Bridge 6.6Gb/s 8.36MB/s 7.9Gb/s 997MB/s 47.8Gb/s 5987MB/s
Baremetal, direct mount (max possible) 7.85Gb/s 981MB/s 19.1Gb/s 2390MB/s 45.1Gb/s 5641MB/s


NFS Client

Dell PowerEdge R610 Dual Intel(R) Xeon(R) X5675 @ 3.07GHz
Network Adapter 40Gb: Mellanox ConnectX-3 CX354A MT27500 Family ConnectX-3
Network Adapter 10GbE: Mellanox ConnectX-2 =MT26448 ConnectX EN 10GigE, PCIe 2.0 5GT/s

NFS Server

Dell r730xd, Dual Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
Network Adapter 40Gb: Mellanox ConnectX-3 CX354A MT27500 Family ConnectX-3
Network Adapter 10GbE: Mellanox ConnectX-2 =MT26448 ConnectX EN 10GigE, PCIe 2.0 5GT/s
Storage: 9x12TB Seagate Exos 12Gb/S SAS 7200 RPM, PERC H730P, mkfs.xfs -f -i size=512 -d su=64k,sw=8 /dev/sdb

[root@hyper0 ~]# omreport storage controller controller=0
 Controller  PERC H730P Mini(Embedded)

ID                                            : 0
Status                                        : Ok
Name                                          : PERC H730P Mini
Slot ID                                       : Embedded
State                                         : Ready
Firmware Version                              :
Minimum Required Firmware Version             : Not Applicable
Driver Version                                : 07.701.17.00-rh1
Minimum Required Driver Version               : Not Applicable
Storport Driver Version                       : Not Applicable
Minimum Required Storport Driver Version      : Not Applicable
Number of Connectors                          : 1
Rebuild Rate                                  : 30%
BGI Rate                                      : 30%
Check Consistency Rate                        : 30%
Reconstruct Rate                              : 30%
Alarm State                                   : Not Applicable
Cluster Mode                                  : Not Applicable
SCSI Initiator ID                             : Not Applicable
Cache Memory Size                             : 2048 MB
Patrol Read Mode                              : Auto
Patrol Read State                             : Stopped
Patrol Read Rate                              : 30%
Patrol Read Iterations                        : 4
Abort Check Consistency on Error              : Disabled
Allow Revertible Hot Spare and Replace Member : Enabled
Load Balance                                  : Not Applicable
Auto Replace Member on Predictive Failure     : Disabled
Redundant Path view                           : Not Applicable
CacheCade Capable                             : Not Applicable
Persistent Hot Spare                          : Disabled
Encryption Capable                            : Yes
Encryption Key Present                        : No
Encryption Mode                               : None
Preserved Cache                               : Not Applicable
Spin Down Unconfigured Drives                 : Disabled
Spin Down Hot Spares                          : Disabled
Spin Down Configured Drives                   : Disabled
Automatic Disk Power Saving (Idle C)          : Disabled
Start Time (HH:MM)                            : Not Applicable
Time Interval for Spin Up (in Hours)          : Not Applicable
T10 Protection Information Capable            : No
Non-RAID HDD Disk Cache Policy                : Unchanged
Current Controller Mode                       : RAID

ID             : 0
Status         : Ok
Name           : Connector 0
State          : Ready
Connector Type : SAS Port RAID Mode
Termination    : Not Applicable
SCSI Rate      : Not Applicable

Virtual Disks
ID                                : 0
Status                            : Ok
Name                              : BOOT
State                             : Ready
Hot Spare Policy violated         : Not Assigned
Encrypted                         : No
Layout                            : RAID-1
Size                              : 558.38 GB (599550590976 bytes)
T10 Protection Information Status : No
Associated Fluid Cache State      : Not Applicable
Device Name                       : /dev/sda
Bus Protocol                      : SAS
Media                             : HDD
Read Policy                       : Read Ahead
Write Policy                      : Write Back
Cache Policy                      : Not Applicable
Stripe Element Size               : 64 KB
Disk Cache Policy                 : Unchanged

ID                                : 1
Status                            : Ok
Name                              : STORAGE
State                             : Ready
Hot Spare Policy violated         : Not Assigned
Encrypted                         : No
Layout                            : RAID-5
Size                              : 89,404.00 GB (95996814032896 bytes)
T10 Protection Information Status : No
Associated Fluid Cache State      : Not Applicable
Device Name                       : /dev/sdb
Bus Protocol                      : SAS
Media                             : HDD
Read Policy                       : Read Ahead
Write Policy                      : Write Back
Cache Policy                      : Not Applicable
Stripe Element Size               : 64 KB
Disk Cache Policy                 : Unchanged

Physical Disks
ID                              : 0:1:0
Status                          : Non-Critical
Name                            : Physical Disk 0:1:0
State                           : Online
Power Status                    : Spun Up
Bus Protocol                    : SAS
Media                           : HDD
Part of Cache Pool              : Not Applicable
Remaining Rated Write Endurance : Not Applicable
Failure Predicted               : No
Revision                        : E001
Driver Version                  : Not Applicable
Model Number                    : Not Applicable
T10 PI Capable                  : No
Certified                       : No
Encryption Capable              : No
Encrypted                       : Not Applicable
Progress                        : Not Applicable
Mirror Set ID                   : Not Applicable
Capacity                        : 11,175.50 GB (11999601754112 bytes)
Used RAID Disk Space            : 11,175.50 GB (11999601754112 bytes)
Available RAID Disk Space       : 0.00 GB (0 bytes)
Hot Spare                       : No
Vendor ID                       : SEAGATE
Product ID                      : ST12000NM0027
Serial No.                      : ZCH06GA7
Part Number                     : Not Available
Negotiated Speed                : 12.00 Gbps
Capable Speed                   : 12.00 Gbps
PCIe Negotiated Link Width      : Not Applicable
PCIe Maximum Link Width         : Not Applicable
Sector Size                     : 512B
Device Write Cache              : Not Applicable
Manufacture Day                 : Not Available
Manufacture Week                : Not Available
Manufacture Year                : Not Available
SAS Address                     : 5000C5009497687D
Non-RAID HDD Disk Cache Policy  : Not Applicable
Disk Cache Policy               : Not Applicable
Sub Vendor                      : Not Available
Available Spare                 : Not Available
ISE Capable                     : No