InfiniBand vs Ethernet Performance
Blowing past 1GbE in your data center, Baremetal vs KVM virtio, NFS rdma vs default 40GbIB vs 40GbE vs 10GbE with SR-IOV and RDMA
Created by: spiderr, Last modification: 05 Jan 2018 (14:59 UTC)
This is not intended to be a comprehensive "real world" test, however it is an apples to apples test using an NFS mount and a simple dd speed test to compare InfiniBand and Ethernet in a virtualized and non-virtualized environment.
Network Adapter 40Gb: Mellanox ConnectX-3 CX354A MT27500 Family ConnectX-3
Network Adapter 10GbE: Mellanox ConnectX-2 =MT26448 ConnectX EN 10GigE, PCIe 2.0 5GT/s
Network Adapter 40Gb: Mellanox ConnectX-3 CX354A MT27500 Family ConnectX-3
Network Adapter 10GbE: Mellanox ConnectX-2 =MT26448 ConnectX EN 10GigE, PCIe 2.0 5GT/s
Storage: 9x12TB Seagate Exos 12Gb/S SAS 7200 RPM, PERC H730P, mkfs.xfs -f -i size=512 -d su=64k,sw=8 /dev/sdb
NFS Export
[root@guest1 ~]# cat /etc/exports
/master1 192.168.72.0/22(rw,insecure,async,all_squash,no_root_squash) 172.16.0.0/22(rw,insecure,async,all_squash,no_root_squash)
[root@baremetal2 ~]# nfsstat -m
/mnt/56ib from 172.16.0.41:/master1
Flags: rw,relatime,vers=4.1,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=172.16.0.14,local_lock=none,addr=172.16.0.41
| Connection | 40Gb IB Mellanox/EMC SX6005 | 40Gb Eth Cisco NX3064 | 10Gb Eth Cisco NX3064 | |||||||
| 16K * 64KB | 16K * 64KB | 16K * 64KB | ||||||||
| Server | Client | Write | Read | w/Cache | Write | Read | w/Cache | Write | Read | w/Cache |
|---|---|---|---|---|---|---|---|---|---|---|
| Baremetal | Baremetal | 3.2Gb/s 409MB/s | 9.8Gb/s 1227MB/s | 47.8Gb/s 5984MB/s | 5.6Gb/s 700MB/s | 8.69Gb/s 1086MB/s | 49.4Gb/s 6184MB/s | |||
| KVM SR-IOV | Baremetal | 4.2Gb/s 529MB/s | 9.6Gb/s 1206MB/s | 47.2Gb/s 5910MB/s | 5.5Gb/s 689MB/s | 9.7Gb/s 1216MB/s | 44.8Gb/s 5602MB/s | 5.0Gb/s 631MB/s | 8.7Gb/s 1088MB/s | 46.4Gb/s 5802MB/s |
| KVM SR-IOV | KVM SR-IOV | 4.5Gb/s 659MB/s | 7.1Gb/s 893MB/s | 40.1Gb/s 5014MB/s | 6.5Gb/s 816MB/s | 7.0Gb/s 884MB/s | 45.3Gb/s 5669MB/s | |||
| KVM Bridge | KVM Bridge | 6.6Gb/s 8.36MB/s | 7.9Gb/s 997MB/s | 47.8Gb/s 5987MB/s | ||||||
| Baremetal, direct mount (max possible) | 7.85Gb/s 981MB/s | 19.1Gb/s 2390MB/s | 45.1Gb/s 5641MB/s | |||||||
Hardware
NFS Client
Dell PowerEdge R610 Dual Intel(R) Xeon(R) X5675 @ 3.07GHzNetwork Adapter 40Gb: Mellanox ConnectX-3 CX354A MT27500 Family ConnectX-3
Network Adapter 10GbE: Mellanox ConnectX-2 =MT26448 ConnectX EN 10GigE, PCIe 2.0 5GT/s
NFS Server
Dell r730xd, Dual Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHzNetwork Adapter 40Gb: Mellanox ConnectX-3 CX354A MT27500 Family ConnectX-3
Network Adapter 10GbE: Mellanox ConnectX-2 =MT26448 ConnectX EN 10GigE, PCIe 2.0 5GT/s
Storage: 9x12TB Seagate Exos 12Gb/S SAS 7200 RPM, PERC H730P, mkfs.xfs -f -i size=512 -d su=64k,sw=8 /dev/sdb
[root@hyper0 ~]# omreport storage controller controller=0
Controller PERC H730P Mini(Embedded)
Controller
ID : 0
Status : Ok
Name : PERC H730P Mini
Slot ID : Embedded
State : Ready
Firmware Version : 25.5.3.0005
Minimum Required Firmware Version : Not Applicable
Driver Version : 07.701.17.00-rh1
Minimum Required Driver Version : Not Applicable
Storport Driver Version : Not Applicable
Minimum Required Storport Driver Version : Not Applicable
Number of Connectors : 1
Rebuild Rate : 30%
BGI Rate : 30%
Check Consistency Rate : 30%
Reconstruct Rate : 30%
Alarm State : Not Applicable
Cluster Mode : Not Applicable
SCSI Initiator ID : Not Applicable
Cache Memory Size : 2048 MB
Patrol Read Mode : Auto
Patrol Read State : Stopped
Patrol Read Rate : 30%
Patrol Read Iterations : 4
Abort Check Consistency on Error : Disabled
Allow Revertible Hot Spare and Replace Member : Enabled
Load Balance : Not Applicable
Auto Replace Member on Predictive Failure : Disabled
Redundant Path view : Not Applicable
CacheCade Capable : Not Applicable
Persistent Hot Spare : Disabled
Encryption Capable : Yes
Encryption Key Present : No
Encryption Mode : None
Preserved Cache : Not Applicable
Spin Down Unconfigured Drives : Disabled
Spin Down Hot Spares : Disabled
Spin Down Configured Drives : Disabled
Automatic Disk Power Saving (Idle C) : Disabled
Start Time (HH:MM) : Not Applicable
Time Interval for Spin Up (in Hours) : Not Applicable
T10 Protection Information Capable : No
Non-RAID HDD Disk Cache Policy : Unchanged
Current Controller Mode : RAID
Connectors
ID : 0
Status : Ok
Name : Connector 0
State : Ready
Connector Type : SAS Port RAID Mode
Termination : Not Applicable
SCSI Rate : Not Applicable
Virtual Disks
ID : 0
Status : Ok
Name : BOOT
State : Ready
Hot Spare Policy violated : Not Assigned
Encrypted : No
Layout : RAID-1
Size : 558.38 GB (599550590976 bytes)
T10 Protection Information Status : No
Associated Fluid Cache State : Not Applicable
Device Name : /dev/sda
Bus Protocol : SAS
Media : HDD
Read Policy : Read Ahead
Write Policy : Write Back
Cache Policy : Not Applicable
Stripe Element Size : 64 KB
Disk Cache Policy : Unchanged
ID : 1
Status : Ok
Name : STORAGE
State : Ready
Hot Spare Policy violated : Not Assigned
Encrypted : No
Layout : RAID-5
Size : 89,404.00 GB (95996814032896 bytes)
T10 Protection Information Status : No
Associated Fluid Cache State : Not Applicable
Device Name : /dev/sdb
Bus Protocol : SAS
Media : HDD
Read Policy : Read Ahead
Write Policy : Write Back
Cache Policy : Not Applicable
Stripe Element Size : 64 KB
Disk Cache Policy : Unchanged
Physical Disks
ID : 0:1:0
Status : Non-Critical
Name : Physical Disk 0:1:0
State : Online
Power Status : Spun Up
Bus Protocol : SAS
Media : HDD
Part of Cache Pool : Not Applicable
Remaining Rated Write Endurance : Not Applicable
Failure Predicted : No
Revision : E001
Driver Version : Not Applicable
Model Number : Not Applicable
T10 PI Capable : No
Certified : No
Encryption Capable : No
Encrypted : Not Applicable
Progress : Not Applicable
Mirror Set ID : Not Applicable
Capacity : 11,175.50 GB (11999601754112 bytes)
Used RAID Disk Space : 11,175.50 GB (11999601754112 bytes)
Available RAID Disk Space : 0.00 GB (0 bytes)
Hot Spare : No
Vendor ID : SEAGATE
Product ID : ST12000NM0027
Serial No. : ZCH06GA7
Part Number : Not Available
Negotiated Speed : 12.00 Gbps
Capable Speed : 12.00 Gbps
PCIe Negotiated Link Width : Not Applicable
PCIe Maximum Link Width : Not Applicable
Sector Size : 512B
Device Write Cache : Not Applicable
Manufacture Day : Not Available
Manufacture Week : Not Available
Manufacture Year : Not Available
SAS Address : 5000C5009497687D
Non-RAID HDD Disk Cache Policy : Not Applicable
Disk Cache Policy : Not Applicable
Sub Vendor : Not Available
Available Spare : Not Available
ISE Capable : No